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ABSTRACT 



Upward Bound is a federal program designed to help 
disadvantaged students prepare to enter and succeed in college. Students 
typically enter the program in their freshman or sophomore year of high 
school and can remain in it through the summer following high school 
graduation. Upward Bound projects provide students with a variety of 
services, including instruction, tutoring, and counseling. This report 
presents findings from a national evaluation of Upward Bound that draws on a 
sample of 67 regular Upward Bound projects. About 1,500 students nationwide 
were in the study group, and about 1,300 other students formed a control 
group. Followup surveys of students and their academic records provided 
information about program effectiveness. Findings show that many students 
remain in Upward Bound only for a short time, and, in fact, the program has 
limited impact on students in high school, and no apparent impact on high 
school graduation. Upward Bound may have some impact on participants' 
postsecondary education experiences, but, since relatively few of the study 
group had reached college age, it is too soon to make definitive statements 
about program impact at the college level. Findings also show that Upward 
Bound has substantial impact on some groups of students, and not others. 
Positive effects were more evident for students who initially had low 
educational expectations and for students with poor performance as they 
entered the program. (SLD) 
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EXECUTIVE SUMMARY 



Upward Bound is a federal program designed to help disadvantaged students prepare to enter 
and succeed in college. Established in 1965, it is the largest federal program other than student 
financial aid programs, to help American high school students attain a postsecondary education. 
Currently, about 44,000 students participate in 563 regular Upward Bound projects around the 
country.' At least two-thirds of each project’s participants must be both low-income and potential 
first-generation college students. Students typically enter the program in their freshman or 
sophomore year of high school and can remain in it through the summer following high school 
graduation. Projects provide students with a variety of services, including instruction, tutoring, and 
counseling. In addition to regularly scheduled meetings throughout the academic year, projects also 
offer an intensive instructional program that meets daily for about six weeks during the summer. 
The vast majority of Upward Bound projects are “hosted” by two- or four-year colleges. 

This report presents findings from the national evaluation of Upward Bound, conducted by 
Mathematica Policy Research (MPR) for the U.S. Department of Education (ED), Planning and 
Evaluation Service.^ This report focuses on the effect of Upward Bound on students during their 
high school years and the first year or two of college.^ Specifically, the report addresses the 
following major research questions: 



• To what extent does Upward Bound further the academic and personal development of 
students during high school? 

• What impact does Upward Bound have on students during their early years in 
postsecondary school? 

• Who benefits most from participating in Upward Bound? 

• What are the typical experiences of students in Upward Bound, for example, in terms 
of how long they participate? Does the amount of time students spend in the program 
influence outcomes? 



'This does not include Math/Science centers or Veterans projects. 

^During the course of the overall evaluation, MPR worked with several subcontractors: Westat, 
Decision Information Resources, Educational Testing Services, and Public/Private Ventures. 



^A previous report (Myers and Schirm 1997) addressed only short-term program impacts on 
students while they were primarily high school freshman and sophomores. 
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SCOPE AND METHODOLOGY 



This report draws on several data sources collected as part of a longitudinal study of Upward 
Bound program effectiveness. The evaluation design initially involved the selection of a nationally 
representative sample of 67 regular Upward Bound projects hosted by two- and four-year colleges. 
Excluded from the study were projects that serve veterans, projects funded through the Upward 
Boimd Math/Science Initiative, and projects hosted by high schools and community-based 
organizations. For information on various project characteristics and practices, we surveyed the 67 
project directors in 1993, as part of a larger survey of Upward Boimd grantees. 

From 1992 to 1994, eligible applicants at these 67 projects completed a baseline questionnaire 
that gathered information about their family backgrounds, attitudes and expectations, and school 
experiences. During the same time period, we also randomly assigned the eligible applicants at each 
project to either a treatment group (Upward Bound) or a control group. Altogether, about 1,500 
students nationwide were assigned to the treatment group and about 1,300 were assigned to the 
control group. We conducted follow-up surveys of both groups in 1994 and 1996 to collect updated 
information on their attitudes, school experiences, and other outcomes. The response rates for these 
surveys were about 97 percent and 85 percent, respectively. We also collected the students’ school 
transcripts in 1994 and 1996 to assess their academic experiences and performance in high school 
and, for those old enough, in college. Finally, project staff reported annually on the participation of 
students in the program. 

To assess most of the program impacts discussed in this report, we compared the average 
outcomes of students in the treatment and control groups. Because the two groups of students were 
essentially indistinguishable at the time of random assignment, any statistically significant difference 
between them in outcomes, such as courses taken in high school or enrollment rates at postsecondary 
institutions, can be attributed to the fact that the one group had the opportunity to participate in 
Upward Bound and the other did not. To analyze the impact of duration in Upward Bound, we 
matched students in the control group with participants who stayed in the program for varying 
lengths of time. 



OVERVIEW OF MAJOR FINDINGS 

When we last collected information from students, many were in their last year of high school 
or had just finished high school and not yet attended college. Because many of the students had not 
had an opportunity to attend college, our most credible findings pertain to students’ high school 
experience. We present findings concerning the college experience because they are suggestive of 
what Upward Bound may achieve in the long run. However, more definitive results concerning 
college access and retention need to wait until data are collected again in late 1998 and 1999. At this 
stage of the national evaluation, several key findings stand out (see Table 1 for a summary): 
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SUMMARY OF FINDINGS ON THE IMPACTS OF UPWARD BOUND 
ON SELECTED STUDENT OUTCOMES 
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• Many students remain in Upward Bound for only a short time. Although most 
participants had the opportunity to remain in Upward Bound for up to three or four 
years, about 35 percent left the program during the first year, and we estimate that at 
least an additional 20 percent will drop out of the program before the end of their senior 
year in high school. The typical participant was exposed to Upward Bound for only 
about 19 months, and remained commonly in the program for one summer and parts of 
two academic years. 

• Upward Bound has limited impacts on students during high school When we looked 
across a variety of outcomes, we found that Upward Bound had impacts on only a few 
outcomes. Roughly two to three years after being selected for Upward Bound, students 
in the treatment group expected to complete slightly more years of education and had 
earned more credits in math and social studies than students in the control group. 
Upward Bound generally had no impacts on students’ in-schools behavior, participation 
in extra curricular activities, grade point average, or credits earned in subjects such as 
English or science. Furthermore, Upward Bound had no impact on high school 
graduation. 

• Upward Bound may have some impact on participants^ postsecondary education 
experiences. Although, Upward Bound had no impact on the chances students attended 
college, students in the treatment group earned more credits from four-year colleges, 
were more likely to receive financial aid, and were more actively engaged in some 
college activities. But since relatively few study participants had reached college age 
and those who did entered Upward Bound during grades 10-12, it is too soon to make 
definitive statements about Upward Bound’s impact on postsecondary outcomes. 

• Upward Bound has substantial impacts on some groups of students and not others. 
Although Upward Bound had small impacts on students as a whole, some groups of 
students received greater benefits. Focusing on the key outcomes of educational 
expectations, courses taken in high school, and high school completion we found that 
(1) students with lower initial educational expectations benefitted substantially more 
than those with higher expectations, (2) boys showed substantially larger impacts than 
girls, (3) Hispanic and white students benefitted more than Afncan American students, 
(4) students who were low-income only or low-income and potential first-generation 
college students showed larger impacts than those who qualified for the program only 
as potential first-generation students, and (5) poorer performing students benefitted 
substantially more than their better performing peers. 



DETAILED FINDINGS 

Program Participation, Completion, Persistence, and Intensity 

About 82 percent of eligible applicants who had the opportunity to participate in Upward Bound 
chose to do so. Among the more common reasons students give for not participating are 



transportation problems, taking a job, not being notified by the project, and time conflicts. Holding 
other factors constant, Hispanics and Asian/Pacific Islanders are more likely to participate than 
whites and African Americans. 

Less than 45 percent of participants continue with the program through the senior year of high 
school.'* Controlling for other factors, students from low-income families are more likely to 
complete the program than their peers who qualify solely by being a potential first-generation 
college student. Also, students who initially expect to obtain at least a bachelor’s degree are more 
likely to complete Upward Bound than those who expect to accumulate less education. 

Focusing on duration rather than completion, we found that over one-third of participants leave 
Upward Bound within the first year and nearly two-thirds leave within two years. The median length 
of participation in the program is 1 9 months. By far the most common reason students give for 
leaving Upward Bound before completing it is to take a job. Holding other factors constant, we 
found, not surprisingly, that the earlier in high school students join, the longer they tend to stay in 
the program. 



Upward Bound is intensive as measured by the number of activities undertaken by students. 
During the academic year, the average student attends 36 academic sessions and 43 nonacademic 
activities; in the summer, the typical participant attends 103 academic sessions and 72 nonacademic 
activities. Over the course of his or her participation in Upward Bound, the average student attends 
a total of 375 sessions, 55 percent of which are academic. 



The Impacts of Upward Bound 

High School Outcomes. Many outcomes concerning students’ experiences were analyzed as 
part of the evaluation; however. Upward Bound had an impact on only a few of them when we 
looked at all Upward Bound students. Upward Bound students had sightly higher educational 
expectations than the control group. Also, the treatment group students earned, on average, slightly 
more nonremedial high school credits in social studies and math-0.1 and 0.2 additional credits, 
respectively. We found no impacts on a host of other outcomes related to high school, including 
misbehavior, participation in school activities, talking with parents about school, or parental 
involvement at school. 

Postsecondary Outcomes. Compared \vith students in the control group, students in Upward 
Bound where just as likely to enroll in a two- or four-year college; however. Upward Bound students 
earned more nonremedial credits at four-year colleges and fewer remedial credits at two-year 
colleges. In addition, the treatment group students were more likely (by margins of 3-4 percentage 
points) to receive various types of financial aid in college, including loans, grants, and work-study 
appointments. Finally, the treatment group participated more often in various college activities, 
including informal contacts with an advisor or faculty member, talking with faculty in their offices. 



‘'This is an upper bound and assumes all current participants will stay in Upward Bound until 
the end of their senior year in high school. 
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participating in a study group with fellow students, and participating in school clubs. These findings 
should be interpreted with caution, however, because only about one-fourth of the students in our 
study had entered college at the time we last contacted them; about one-third were still in high 
school. A more comprehensive analysis of how Upward Bound affects students’ postsecondary 
education pursuits will be possible in 1999, after we next survey the study participants and again 
collect college transcripts. 

Subgroup Differences. Upward Bound had large impacts on students specific characteristics, 
such as students with lower educational expectations, boys, and white and Hispanic youth. Some 
highlights of our analysis of subgroup differences include the following: 



• Among students with initially lower educational expectations (that is, students who 
expect to complete less than a bachelor’s degree). Upward Bound students earned an 
average of almost three high school credits more than those in the control group and 
were less likely to drop out of high school than those who did not participate in Upward 
Bound. We found no such difference among students who expected to complete at least 
a bachelor’s degree. Similar results were found for college enrollment: students with 
lower expectations who were in the treatment group were more likely to attend college 
than students in the control group and we found no impacts for those with higher 
educational expectations. 

• Boys in Upward Bound earned almost two more high school credits than did boys in the 
control group; girls selected for the program did not earn more total high school credits 
than girls in the control group. Boys in Upward Bound also were less likely to drop out 
of high school and were more likely to attend four-year colleges than boys in the control 
group. Upward Bound had no impact on college enrollment for girls. 

• Whites and Hispanics in the treatment group earned almost two more credits in high 
school than similar students in the control group, but there was no such difference 
among African American students. Furthermore, white and Hispanic Upward Bound 
students were less likely to drop out of high school than similar students in the control 
group; there was no impact on dropping out for African American students. In addition, 
whites who were in Upward Bound were more likely to enroll in college (two- and four- 
year colleges combined) than similar students in the control group and Hispanics in 
Upward Bound attended four-year colleges at a higher rate than those who were in the 
control group. 

• Upward Bound students who qualified for the program solely because of low-income 
earned an average of about two credits more in high school than similar students not 
selected for the program and were less likely to drop out of high school. Those who met 
both the low-income and first-generation criteria and were selected earned about one 
credit more and were somewhat less likely to drop out of school than those not selected. 
We found no difference among those who were eligible only as potential first- 
generation college students. 
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• Poorer-performing students (a categorization based on both grade point average and 
core academic credits earned in 9th grade) who were in Upward Bound earned, on 
average, almost two more credits in high school than those in the control group. We 
found no similar difference among better-performing students. In addition. Upward 
Bound reduced the chances that poorer-performing students would drop out of high 
school; the program had a small impact for those less at risk. Also among the poorer- 
performing students, those who were in Upward Bound attended four-year colleges at 
a higher rate than similar students in the control group. Upward Bound had no impact 
on college attendance for better-performing students. 



The Impact of Duration in Upward Bound on Educational Outcomes 

Longer exposure to Upward Bound was associated with greater program impacts. Students who 
entered Upward Bound in the 9th grade and remained in for more than two years had higher 
educational expectations and earned about three more credits in core academic subjects than similar 
nonparticipants. Similarly, students who began the program as 10th graders and persisted for more 
than two years earned about two more core academic credits than their counterparts. As for 
postsecondary outcomes, students who started Upward Bound as 10th graders and remained in 
Upward Bound for 13-24 months were more likely (by a margin of 14 points) to go on to a four-year 
college and also earned almost six more nonremedial credits at four-year schools. Comparison of 
the experiences of students who completed Upward Bound and noncompleters suggests that 
continuing in the program until the end of the senior year of high school generally has no impact on 
students’ outcomes. 



CONCLUSIONS AND IMPLICATIONS 

Although Upward Bound generally has few and small impacts on students during their high 
school years, and it is too soon to tell for certain how the program affects students in terms of college 
attendance and completion, our findings demonstrate that Upward Bound can have large impacts for 
some groups of students. Most notably, the program appears more helpful to students wdth lower 
initial educational expectations, students with poorer academic performance as high school 
freshmen, and those who remain in the program for at least two years. These findings suggest that 
if the U.S. Department of Education (ED) or individual projects want to increase the impact of the 
program, they may want to consider action in two areas: retention and student selection. 



Improve Student Retention 

As mentioned above, only about one-third of the Upward Bound participants in our study had 
been in the program for more than two years at the time of our last data collection, and about 40 
percent had stopped participating within one year. Yet, the longer students remain in the program, 
the more they appear to benefit in terms of both high school and college outcomes. Therefore, one 
obvious way to increase overall program effectiveness is to improve student retention. 
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While a variety of retention improvement strategies may be worthy of consideration, a logical 
approach would be to address a common reason students leave the program before high school 
graduation--to get a job. One way to retain participants longer would be for projects to offer students 
more job opportunities in the summer. Such a strategy has a number of potential drawbacks, 
including a reduction in academic instruction. In addition, to maintain the academic character and 
focus of Upward Bound, an important goal would be to arrange work experiences that complement 
a project’s design and curriculum. Nonetheless, although not easy to accomplish, improving 
program retention rates through greater work opportunities could result in greater impacts on student 
outcomes. 



Serve More At-Risk Students 

Students who enter Upward Bound expecting to complete less than a bachelor’s degree benefit 
more from the program than their peers with higher expectations, but they account for only about 
one-fifth of all students served. Furthermore, students who are more at risk of academic failure as 
freshman benefit more from Upward Bound than students less at risk. Increasing the proportion of 
9th graders served who have lower educational expectations or have poorer records of academic 
performance could substantially boost the average impact of Upward Bound on a variety of 
important outcomes. 

How might project staff attract such students? Three possible strategies include (1) recruiting 
more students with grade point averages in the C or D range as high school freshmen; (2) recruiting 
more students who have earned relatively few credits in the core academic subjects of English, math, 
science, social studies, and foreign language; and (3) working with target school personnel to 
identify students who have the potential to complete a four-year degree, but who plan to pursue some 
other postsecondary objective or not to attend a postsecondary school at all. 

Trying to change the mix of students served may be controversial among some project staff and 
may have an impact on projects’ performance measures because they would be serving a group of 
students whom we would expect to be less likely to attend college. Efforts to do so should be 
monitored carefully to guard against possible adverse effects and current accountability requirements 
should be aligned with this objective. If ED is interested in exploring this idea, one option would 
be to set up demonstration projects that enroll different proportions of students with lower 
expectations, and compare impacts over time. 
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I. INTRODUCTION 



A college degree is becoming increasingly critical to success in the workforce. College 
graduates have substantially lower unemployment rates and substantially higher incomes than 
individuals with less education. Students from disadvantaged backgrounds have historically been 
much less likely than their peers to complete a college education. 

A major federal effort to address this concern is Upward Bound, a program designed to help 
disadvantaged students prepare for college by providing supplemental academic and support 
services. In FY 1998, with federal funds of more than $170 million, the program served about 
42,000 students in 566 projects nationwide. Learning more about Upward Bound program 
effectiveness will not only help to improve Upward Bound, but also potentially hundreds of other 
precollege programs operating throughout the country. Ultimately, policy makers and program 
operators must know the extent to which their efforts actually improve the educational outcomes 
of economically disadvantaged students. 

This report summarizes the impacts that Upward Bound has on students through their high 
school years and, for some students, their first one or two years in college. It represents one piece 
of an ongoing, long-term study that will track students through the college years. This is the final 
report from the first five-year phase of a national evaluation of Upward Bound, sponsored by the 
U.S. Department of Education (ED) and conducted by Mathematica Policy Research, Inc. (MPR) 
and its subcontractors: Westat, Decision Information Resources, Educational Testing Service, and 
Public/Private Ventures.' 

'Previous reports have addressed the short-term impact of Upward Bound (Myers and Schirm 
1997) and programs offered, students served, and operational issues (Moore 1997). The second five- 

(continued...) 



A. THE CONTEXT AND PURPOSE OF UPWARD BOUND 

In the mid-1960s and early 1970s. several well-publicized studies called attention to the poor 
academic preparation and low educational attainment of low-incorne youth (see. for example. 
Coleman et al. 1966, Jencks et al. 1972, Mosteller and Moynihan 1972). Concern about major 
disparities in the educational outcomes of children from different income groups resulted in the 
creation of some major federal education programs as part of the War on Poverty, including Head 
Start for young children and the Upward Bound program for high school-age youth. The purpose 
of Upward Bound is to help prepare disadvantaged high school students to enroll and succeed in 
college. 

Despite the recent improved academic achievements of low-income and minority students, 
substantial gaps still exist between low-income and better-off students. For example, between 1972 
and 1995, the percentage of high school graduates who were enrolled in college during the October 
or after graduation rose from 26 to 34 percent among students from low-income families, but at the 
same time the enrollment rate among middle incorne students rose from 45 to 56 percent, and among 
high income students it went from 64 to 83 percent." Even taking a longer view of postsecondary 
educational attainment, students of lower socioeconomic status do not fair as well as their more 
advantaged peers. Among high school sophomores expected to graduate from high school in 1982, 
only seven percent of those from the lowest socioeconomic quartile had earned a bachelor's degree 



'(...continued) 

year phase of the evaluation, which will end in 2002, will result in reports addressing the longer-term 
impacts of the regular Upward Bound program, common practices of highly effective Upward 
Bound projects, and the impacts and benefits of Upward Bound Math/Science projects. 

^U.S. Department of Education, National Center for Education Statistics. The Condition of 
Education, 1997. Indicator 8. NCES 97-388. Washington, DC: U.S. Government Printing Office, 
1997. 
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or higher by 1 992. compared with 22 percent of those in the middle two quartiles. and 5 1 percent 
of those from the highest quartile.^ Thus, more than 30 years after its creation, the Upward Bound 
program seems to be needed as much as ever. 



B. MAJOR FINDINGS FROM THE NATIONAL EVALUATION 
1. How Upward Bound Operates Today 

There are more than 550 Upward Bound projects currently operating around the nation. The 
key elements of these projects are the students, the services, and the staff: 



• Students. At least two-thirds of the students served by each Upward Bound project 
must be both low-income and potential first-generation college students; the remainder 
must meet one of these criteria. Most participants join the program in their early years 
of high school. Furthermore, most of the students who currently apply for Upward 
Bound are educationally motivated, have few behavioral or disciplinary problems, and 
earn mostly B's or C's in school. 

• Services. Upward Bound, unlike many other pre-college programs, provides students 
with an opportunity to generally participate in the program for up to four years. Upward 
Bound offers students an intensive program that meets regularly during the academic 
year and the summer. Projects typically provide a wide variety of services, including 
traditional academic instruction, tutoring, mentoring, counseling, career planning, 
cultural programs, and college planning activities. In addition to regular meetings 
during the academic year, programs also offer an intensive instructional program in the 
summer, typically lasting about six weeks, and at almost 90 percent of projects the 
summer session includes a residential component during which students live on a 
college campus to help simulate the college experience. And in many (85 percent) 
Upward Bound projects after participants graduate from high school, they can 
participate in a summer “bridge” program that is intended to smooth the transition to 
college. 

• Staff. Project staff generally hold a bachelor’s or graduate degree. Many staff are from 
the same racial/ethnic group as the majority of their participants, which was often noted 



^U.S. Department of Education, National Center for Education Statistics. The Condition of 
Education, 1996. Table 26-2. NCES 96-304. Washington, DC: U.S. Government Printing Office, 
1996. 
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by project staff and others as important since staff members become role models for the 
students. 



To provide a fuller picture of program operations, following are some of the major findings from 
our comprehensive assessment of the Upward Bound program (Moore 1997), which drew on 
national surveys of grantees and target schools as well as case studies of 20 projects: 



• Upward Bound projects typically offer a rich and challenging academic progranu 
Projects’ heavy emphasis on academic preparation for college is evident in three areas: 
(1) the number of courses offered--50 percent offer more than 17 academic courses in 
the summer session and more than 10 academic courses during the school year; (2) the 
nature and content of the courses—fewer than one-third of projects focus on remedial 
instruction and most projects offer courses that reflect a fairly traditional precollege 
prepatory curriculum; and (3) course requirements— 80 percent of projects require 
students to complete at least six courses, typically comprising either a traditional 
curriculum focused on reading, writing, algebra and geometry, or one with a heavy focus 
on science and more advanced math. 

• Upward Bound is primarily student-centered; parent- and school-centered activities 
are secondary. Project operations are primarily intended to influence students directly, 
not to change the student’s family or school. While virtually all projects provide some 
opportunities for parental involvement, the emphasis is sometimes limited. In fact, 
parents may desire a closer connection to the programs. 

• Projects reported high college enrollment rates for seniors. Of the students who 
remain in the program through their senior year of high school, about 85 percent enroll 
in college the next fall according to project reports. Furthermore, projects reported that 
about two-thirds of their graduates enroll at a four-year college. 



2. Previous Results Based on Students' Early Involvement in Upward Bound 

Because of the timing of data collection, our previous report (Myers and Schirm 1997) was only 
able to describe the short-term impact of Upward Bound on students, while most students were still 
in the early high school years. The major findings of that report include the following: 



Upward Bound had early positive impacts on students’ academic course taking and 
educational expectations. Students in Upward Bound who entered the program while 
high school sophomores or later earned, on average, about one credit (Carnegie unit) 
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more in core academic subjects—English, science, math, social studies, and foreign 
languages-than did those who were in the control group. In addition, in the two to three 
years after they applied for the program, both students in Upward Bound and their 
counterparts in the control group typically lowered their expectations for how many 
years of schooling they would complete; however, the decline was smaller for those who 
were in Upward Bound than for those who were not. 

• Students with lower educational expectations benefit most from Upward Bound. 
Among students who initially did not expect to complete a college degree, those who 
were selected for Upward Bound earned about three more academic credits than those 
who did not participate. In comparison, among students with higher initial educational 
expectations, students in Upward Bound earned only about .5 more academic credits 
than those in the control group. We also found a positive impact among the lower 
expectations group, but not the higher expectations group, on parents' expectations for 
the amount of schooling their children would complete, as reported by the students. 

• Hispanic students initially benefit more from Upward Bound than do students from 
other racial/ethnic groups. Among the three largest racial/ethnic groups in Upward 
Bound-African Americans, Hispanics, and whites-Hispanics consistently experienced 
the largest benefits from participating in the program. For example, Hispanic Upward 
Bound students earned about two credits more in core academic subjects than did 
Hispanic students in the control group. In comparison, African American and white 
participants earned less than .5 credits more than nonparticipants from those 
racial/ethnic groups (differences that were not statistically significant). 

/ 

The earlier report also found that Upward Bound participants have a high early attrition rate. 
An estimated 37 percent of students who entered Upward Bound left the program within the first 12 
months, especially to take jobs. Moreover, longer participation in Upward Bound was associated 
with earning more high school credits than those who left the program sooner. 



C. KEY TOPICS ADDRESSED IN THIS REPORT 

This report builds upon the results of our earlier work; it addresses many of the same issues, and 
is based on the same students, but adds data collected two to three years later. The general question 
underlying this report is how Upward Bound affects students as they continue to progress in their 
schooling and other areas. Specifically, the report addresses the following major research questions: 





• To what extent does Upward Bound further the academic and personal development of 
students during high school? 

• Does Upward Bound have an impact on college access and retention? 

• Do some groups of students benefit more from Upward Bound than other students? 

• What are the typical experiences of students in Upward Bound, for example, in terms 
of how long they participate? Does the amount of time students spend in the program 
have an influence on various outcomes? 

D. RESEARCH DESIGN 

To address the questions listed above, MPR has implemented the first rigorous evaluation of 
Upward Bound since the late 1970s. The key feature of the research design is the use of random 
assignment to select students for Upward Bound in a nationally representative sample of Upward 
Bound projects. The random assignment design allows us to compare education-related outcomes 
of students in the program to outcomes of students without access to the program. 

1. Random Assignment v 

From the list of regular Upward Bound projects, we randomly selected 70 for the program 
impacts study."* However, 1 1 of the selected projects could not participate or had to be ruled out for 
various reasons. For example, some were defimded by ED during the 1991-92 grants competition; 
others did not plan to recruit new students for the 1992-93 school year; and some had too few 
applicants to accommodate random assignment. We replaced eight of these 1 1 projects with similar, 
randomly selected projects, and ended up with 67 Upward Bound projects. 



"’The sample represents Upward Bound projects that are (1) located in the 50 states or the 
District of Columbia; (2) hosted by a postsecondary institution; (3) mature, having operated for at 
least three years by October 1992; and (4) not dedicated to serving only students with physical 
disabilities. 
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During the 1992-93 and 1993-94 school years, we randomly assigned eligible applicants from 
each site to either a treatment group, which was invited to participate in the program, or a control 
group, which was not invited to participate.^ Eligible applicants were defined as students the 
projects had recruited who met both the federal requirements (concerning income or first-generation 
status), as well as any project-specific criteria for participation. All the projects had more applicants 
than openings, and all ended up serving the same number of students they would have served under 
their usual selection procedures.* * We implemented the random assignment slowly over 14 months 
so that projects could use their normal recruiting procedures and enroll students following their 
normal enrollment schedules. Nationwide, the random assignment process resulted in a treatment 
group of about 1,500 students and a control group of about 1,300. A more detailed description of 
our random assignment procedures is presented in Myers et al. (1993). 

The random assignment design allows us to calculate estimates of program impacts that are free 
of selection biases. The only difference between the treatment and control groups is that the former 
was offered the opportunity to participate in Upward Bound; otherwise, the two groups are 



*We later learned that project directors had allowed a total of 29 control group members, out of 
more than 1 ,300 students in the control group, to participate in Upward Bound. Although these 
students continued to receive program services, we maintained their original status as members, of 
the control group. While this results in a trivially small underestimate of program impacts, to 
consider these students members of the treatment group or remove them from the analysis sample 
would violate the random assignment design and reduce the study's internal validity. 

*To accommodate project wishes concerning the composition of the participants— such as sex 
or racial/ethnic balances— we used stratified random sampling when selecting the treatment and 
control groups. In addition, before random assignment, we asked project directors to rate each 
applicant as either most likely to be selected under normal selection procedures, somewhat likely 
to be selected, or least likely to be selected. Later analyses of these three groups showed that they 
had few differences in their observable characteristics and few differences in who benefitted most 
from the program. These findings suggest that random selection of students meeting the federal and 
project-specific selection criteria did not substantively alter the kinds of students projects normally 
serve. 
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statistically equivalent (see Myers and Schirm 1997). Comparing the outcomes for the two student 
groups indicates, for all applicants, the impact of being offered a chance to participate in Upward 
Bound. However, not all students randomly selected for the treatment group chose to enroll in the 
program.’ By making statistical adjustments, we can estimate the impacts of program participation. 
In subsequent chapters, we provide both types-of estimates. The sizes of the two types of estimates 
are generally close, but in our summary of findings we focus on impacts for all students. 

2. Data Collection 

The analyses in this report are based on information from students, their schools, and the 
Upward Bound projects they applied to or participated in. Virtually all sample students completed 
a baseline written questionnaire at the time of application, prior to random assignment. In the spring 
of 1 994 we conducted a first follow-up survey of the students, by telephone, achieving a response 
rate of 97 percent. In 1996 we conducted a second follow-up telephone survey, achieving a response 
rate of about 85 percent. Each of these surveys addressed a wide variety, of topics, including the 
students' background; experiences related to school, supplemental services, and employment; and 
their plans for the future. With the students' permission, we also collected their official school 
transcripts in 1994 and 1996, to determine the number and type of courses they had taken. At the 
start of our study, the directors of the 67 participating projects completed a written questionnaire 
about various aspects of program operations. 



’The “no-show”. rate for the entire treatment group and for various subgroups (for example, sex 
and grade level) are shown in Chapter III. 
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E. PHASE II OF THE NATIONAL EVALUATION OF UPWARD BOUND 



The U.S. Department of Education (ED) has awarded a contract to MPR to follow the current 
sample of students for five more years to assess the program's impact on a series of college related 
outcomes, such as access to, retention in, and completion of college; selectivity of the colleges 
attended; college major; and employment related outcomes. Reports describing program impacts 
will be available in 1999 and 2001. 

ED has also funded two studies linked to the ongoing evaluation. The first is an assessment of 
the practices used by nine Upward Bound projects that have potentially large impacts on key 
outcomes, such as educational expectations, emd credits earned in core high school subjects. MPR 
staff emd its subcontractors will visit these projects twice: once during the school year and another 
time during the summer program in 1998. The final report describing the findings from the case 
study visits will be available in early 1999. The second new study is an evaluation of the effects of 
Math/Science centers (MSCs), funded under the Upward Bound Math/Science Initiative, on 
students’ outcomes, such as college major. This evaluation will compare the experiences of a sample 
of MSC participants with those of similar students who are already part of the national evaluation. 
Reports describing the effects of MSC participation on student outcomes will be available in late 
1999 and 2001. 

F. ORGANIZATION OF THIS REPORT 
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The remainder of this report is organized into four chapters. Chapter II describes students’ 
participation and persistence in Upward Bound. Chapter III addresses the impact of Upward Bound 
on students in terms of educational expectations, courses in high school, college attendance, and 
other important outcomes. We also analyze outcomes for different groups of students, such as 
students with relatively lower and higher initial educational expectations, boys and girls, and various 
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racial/ethnic groups. Chapter IV discusses how the length of time students spend in Upward Bound 
affects various secondary and postsecondary outcomes. Chapter V presents our conclusions and a 
discussion of potential policy implications. In addition, several appendices present details 
concerning the analyses conducted for this report. 



II. THE UPWARD BOUND EXPERIENCE: PARTICIPATION, 
COMPLETION, PERSISTENCE, AND INTENSITY 



In this chapter, we examine the Upward Bound experiences of the roughly 1 .500 eligible 
applicants randomly selected between late 1992 and early 1994 for participation in the program. 
Specifically, we answer the following questions: 

• What fraction of eligible applicants choose to participate in the program when given the 
opportunity? Which ones participate? Why do some not participate? 

• What fraction of participants remain in the program until they graduate from high 
school? Which ones are more likely to remain in the program that long? 

• How long do participants typically remain in the program? Do some remain in the 
program longer than others? Why do some participants leave early? 

• In what activities do participants engage while in the program? 

• Are some projects better able to retain students than other projects? 

The answers to these questions are important for gauging the effectiveness of the program in 
attracting and retaining students. They also provide a context for interpreting the impact results 
presented in this report, indicating how much of a “dose" of Upward Bound the program’s 
participants typically get. Finally, the findings presented in this chapter are a critical input to our 
analysis of the effects of duration on student outcomes. The key results from our analysis of 
students’ Upward Bound experiences are: 



• Out of the roughly 20,000 eligible applicants offered the opportunity to participate in 
Upward Bound each year, about 16,400 enter the program, and 10,600 remain in 
Upward Bound for at least one year, as depicted in Figure II. 1. About 7,200, at most, 
ever complete the program, remaining in Upward Bound until high school graduation. 
The typical Upward Bound student participates in the program for 19 months and 
attends nearly 400 sessions of Upward Bound academic and nonacademic activities. 
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FIGURE III 

A COHORT’S PATH TO COMPLETING UPWARD BOUND 




1 1 Participate for 
at least 1 Year 




7 Participate until 
High School 
Graduation 



• Some groups of students participate and persist in Upward Bound longer or complete 
the program at higher rates than other groups. For example, younger participants stay 
in longer but are less likely to complete the program than older participants. Students 
with lower educational expectations are as likely to enter Upward Bound but have lower 
persistence and are less likely to complete the program than students with higher 
expectations. First generation only students tend to enter and complete Upward Bound 
at lower rates than low-income students. Boys and girls have similar patterns of 
participation. 

• There are no consistent differences among the participation patterns of students from 
different types of projects, such as small and large projects. 



The principal data zinalyzed in this chapter were collected in the late summer and fall of 1 994, 
1995, and 1996. For each Upward Bound student in the study, program staff provided information 
on the periods and activities in which the student participated, indicating when the student started 
and permanently left the program, as well as any temporary withdrawals. Although the Upward 
Bound students in the study were randomly selected, comparisons of students who would normally 
be selected for Upward Bound with students who would not normally be selected (as identified by 
project directors) demonstrate that our results reflect the typical experiences of Upward Bound 
students. Our results are not affected by random assignment having replaced the program’s usual 
selection procedures.' 



A. PROGRAM PARTICIPATION 

More than one in six eligible applicants (18 percent) offered the opportunity to peirticipate in 
Upward Bound never do so. To place this in perspective, the Job Corps program which serves a 



'Even though it does not matter for our results how slots are allocated to eligible applicants, it 
may matter substantially how the pool of eligible applicants is formed. Our results pertain to the 
types of eligible students who are recruited for and apply to the program and whose characteristics 
were described earlier in this report. If Upward Bound projects began recruiting a substantially 
different type of eligible student, such as students from the bottom of the grade distribution, the new 
recruits who apply and are offered admission to the program may have participation, completion, 
and persistence rates that are very different from the rates of current Upward Bound students. 



more disadvantaged group of youth, has a no-show rate that is somewhat higher than Upward 
Bound— about 25 percent. Students give a wide range of reasons for not participating, with no one 
reason given very frequently, as shown in Table II. 1.- The nonparticipation rate is 18 percent in spite 
of the lengthy and rigorous recruiting and application process during which program staff and each 
student can assess the student’s interests and motivation. This process gives a student ample 
opportunity to determine that Upward Bound may not be the program for him or her and to be 
removed from consideration for the program. Nevertheless, a substantial fraction of students 
completing the process choose not to take advantage of Upward Bound when it is offered. 

The chances of participating in Upward Bound are associated with only a few observed student 
and project characteristics: race, scheduled time of entry into the program, and being in the sexual 
minority within a project. The relationships between other characteristics and participation rates are 
weak or, if seemingly strong, not statistically significant. For some key subgroups of students, the 
estimated participation rates are: 81 percent for females, 84 percent for males, 78 percent for Afncan 
Americans, 83 percent for whites, 9 1 percent for Hispanics, 82 percent for students who expected 
to complete a four-year college degree, and 81 percent for students who did not expect to complete 
four-year college degree. 

These subgroup participation rates are “unadjusted," reflecting differences between subgroups 
in many characteristics. Thus, the three percentage point difference between the participation rates 
for females and males is attributable to not only the difference in and influence of sex on 




^The results presented in this section are essentially the same as the results presented in Myers 
and Schirm (1997), except here we consider the effects on participation of a broader range of student 
and project characteristics. In the next two sections, we are able to substantially refine our earlier 
analyses because much additional data on participation have become available. 
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TABLE II. 1 



MOST COMMON REASONS FOR NOT PARTICIPATING IN UPWARD BOUND 



; Reason ;'x '; , • . • Percentage Giving Reason . • 


Transportation problems 


20 


Took a job 


18 


Program did not contact me 


18 


Time conflict 


15 


Family issues 


11 



participation, but also the differences between males and females and the influences of race, 
educational expectations, and many other characteristics. 

Table II.2 displays “adjusted" participation rates. These rates allow us to contrast the 
participation rates of students who have all the same characteristics except for one, such as sex or 
race.^ 

Among racial groups, Asians and Hispanics have the highest participation rates at 90 percent 
or higher according to Table II.2. The participation rates for African American, white, and Native 
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^For each characteristic, we have identified by a “**” a reference category, which is the most 
common attribute eimong Upward Bound students. Considering the relationship between sex and 
participation rates, for example, we see in Table II.2 that the reference category is female because 
more Upward Bound students are female. The adjusted participation rate of 72 percent for males 
shows the frequency of participation that we predict would occur among males if they had the same 
distribution of other characteristics (e.g., race and educational expectations) as females. Because 
females already have that distribution of other characteristics without any adjustment, the adjusted 
and unadjusted participation rates for females are the same--81 percent, or 9 percentage points higher 
than the adjusted rate for males. As we noted earlier, males actually have a higher unadjusted 
participation rate than females (86 versus 81 percent), indicating that males are more likely than 
females to have other characteristics associated with high participation rates, in particular, males are 
much more often in the (sexual) minority in their projects. Imposing a common distribution of all 
the other characteristics listed in Table II.2, we have “adjusted” for the effects on participation of 
extraneous differences between males and females to isolate the relationship between sex and the 
Upward Bound participation rate. The adjusted participation rates in Table II.2 were obtained using 
a logistic regression model, which is an appropriate specification when the phenomenon of interest 
has two outcomes (e.g., participate/did not participate). Each “p-value” in Table II.2 indicates the 
lowest significance level at which we would reject the hypothesis that there are no differences among 
categories of a characteristic. Generally, we are regarding a relationship as significant if the 
associated p-value is under 0.10. 
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TABLE II.2 



ADJUSTED participation RATES 



■" Characteristic Participation Rate (%) . p^valne for %*• 



Female** 

Male 



0.15 

81 

72 



Race 

African American** 

Hispanic 

Asian 

White 

Native American 

Scheduled entry into Upward Bound 
During 8th grade 
Summer after 8th grade 
During 9th grade 
Summer after 9th grade** 
During 1 0th grade 
Summer after 1 0th grade 
During I Ith grade 
Summer after I Ith grade 



0.09 

78 
90 
93 

79 
78 

0.09 

90 

86 

71 
83 
88 

72 
82 
63 



Low-income/first generation 

Both low-income and first generation** 
Low-income only 
First generation only 



0.1 1 

82 

95 

77 



Project director’s rating 
Most likely to serve** 
Somewhat likely to serve 
Least likely to serve 



0.25 

85 

78 

87 



College plans 

Plan to complete a degree** 

Do not plan to complete a degree 



0.89 

82 

82 



Student in racial minority within project 
yes 

no** 



0.13 

87 

80 



Student in sexual minority within project 
yes 

no** 



0.08 

89 

80 




Project host institution type and control 
Four-year public institution** 
Four-year private institution 
Two-year institution 



0.43 

79 

84 

78 
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TABLE 11.2 (continued) 



l^aractemtic 


Participation Rate (%) 


: - p-value ^ 


Project is racially diverse (dominant race has less than 




0.1 1 


60 percent of students) 


yes 


87 




no** 


80 




Project is racially homogeneous (dominant race has 




0.91 


100 percent of students) 


yes 


82 




no** 


82 




Project is sexually diverse (dominant sex has less than 




0.26 


60 percent of students) 


yes 


83 




no** 


78 




Academic requirements 




0.49 


Math/Science** 


77 




Foundational 


76 




Structured 


75 




Unstructured 


82 




Uncategorized 


88 





*p-values corresponds to the probability of rejecting the hypothesis that one or more of the participation rates differ 
from the others in the group by chance alone. 

**The reference attribute. 

SUOl 18 (5/1 1/98, 15:44) and PRED9A.WK4. 
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American students, who together constitute about three quarters of Upward Bound students, are all 
just under 80 percent, significantly different from the rates for Asians and Hispanics/ 

Perhaps the most puzzling result from our analysis of participation is the relatively low 
participation rate for students selected to enter the program during ninth grade. Furthermore, for 
each of the other three grades, there is a lower participation rate— sometimes a much lower rate— for 
students scheduled to start in the summer than for students scheduled to start in the academic year. 
This pattern probably reflects the coincidence of two phenomena: (1) the Upward Bound program 
is more intensive and more demanding during the summer and (2) there are more conflicting 
activities, such as jobs and vacations, confronting the student during the summer. As expected, the 
summer “drop off’ becomes more pronounced as students age. Because ninth grade is the first year 
of high school for many students, perhaps it is the many academic and social pressures associated 
with entering high school that explain why students who are supposed to enter the program during 
the ninth grade academic year participate at a lower rate than students who are supposed to enter the 
summer after ninth grade and, indeed, at a lower rate than any group except students scheduled to 
start the summer after eleventh grade. ^ 



‘‘The Hispanic-Native American difference is significant at the 0.1 1 level. For multiple category 
characteristics like race, the results of pairwise comparisons such as Hispanic-Native American or 
Asian-African American are not displayed in Table II. 2, although they are discussed when 
interesting. 




*If we set aside the possibly anomalous result for the ninth grade academic year, we see that 
academic year participation rates are fairly stable, while summer participation rates are lower and 
decline with age. Sample sizes are not sufficiently large for even some of the bigger differences to 
be significant. However, students scheduled to enter before ninth grade or during tenth grade do 
have significantly different participation rates from students scheduled to enter during ninth grade 
or during the summers after tenth and eleventh grades. 
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In addition to the differences in participation rates by race and scheduled time of entry into the 
program, we find other significant differences: 



• Students who are potential first generation college, whether low-income or not. are 
much less likely to participate than students who are low-income only. 

• Students who will be in the minority in their projects— by either race or sex— are more 
likely to participate than students who will not.* That such students are even willing to 
apply under such circumstances may suggest that they are more motivated to overcome 
obstacles like the prospect of “not fitting in.” Alternatively, they may be more highly 
recruited by project staff seeking to diversify their programs. 



We found that a wide range of other observable student and project characteristics such as 
educational expectations and the type and control of the program’s host institution do not seem to 
be associated with the likelihood of entering the program.’ 

The most important finding of no significant differences among participation rates is for project 
directors rating. Prior to the random assignment of students, each Upward Bound project director 
rated eligible students as “most likely to serve,” “somewhat likely to serve,” or “least likely to serve.” 
The “most likely to serve” students are essentially the students the project directors would have 
selected in the absence of random assignment and are, therefore, the kinds of students who would 
typically be given the chance to participate in Upward Bound. We found that the participation rate 
for these students is not significantly different from the participation rates for students in the other 



*The difference in participation rates between students who will be in the racial minority and 
students who will not is not statistically significant at the level that we are using for most 
comparisons (0.10), although it is significant at a slightly higher level (0.13). The larger estimated 
difference in participation rates between students who will be in the sexual minority and students 
who will not is significant at the 0.08 level. 




’In preliminary analyses, we also found no significant association between student participation, 
completion, or persistence rates and the size of projects or their location (urban/rural). Therefore, 
we excluded these two project characteristics from the analyses presented in this chapter. 
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two groups. Thus, the estimated participation rate for the students who were selected at random for 
Upward Bound is probably typical for the program and not a consequence of offering Upward Bound 
to students who would not normally be selected. 

B. PROGRAM COMPLETION 

Fewer than half of the students (44 percent) who start Upward Bound complete the program. 
By "complete,” we mean that a student was still participating in May of the year that the student 
graduated from high school.*’ To calculate the fraction of students completing the program, we only 
counted students who had graduated from high school before August 1 996, the last month covered 
by our data on participation. Because many (nearly 60 percent) of the students who will graduate 
later had already permanently dropped out of Upward Bound before August 1996, the “final” 
completion rate that will be obtained after future rounds of data collection will fall within a range 
of 31 percent to 44 percent.'® 



*For the small number of students graduating between terms of the academic year, a program 
completer was still participating in the month of graduation. 

’This definition ignores participation in projects’ summer bridge programs. Hence, it is a liberal 
definition, overstating completion rates for students in projects where the bridge component is an 
important element of the Upward Bound program. 



'°If all of the students who were still participating in Upward Bound in August 1996 complete 
the program, which is unlikely, the final completion rate will be 44 percent. If none of those 
students complete the program, the completion rate will drop to about 3 1 percent. Nearly all of the 
students who had not graduated before August 1 996 and were still in Upward Bound had entered the 
program relatively “early” (before the end of ninth grade). Our calculated minimum and maximum 
overall completion rates suggest that the completion rates for these early entrants will likely fall 
below the rates for the later entrants. Such an outcome would imply that the effect of getting a 
student started early is not strong enough to overcome the effect of being exposed longer to 
competing opportunities—for example, jobs and extracurricular activities--and other obstacles to 
completion. 
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Although program completion rates do not seem to vary by most student and project 
characteristics, there are large differences associated with time of entry into Upward Bound, low- 
income/first generation status, and college plans. For some key subgroups of students, the estimated 
completion rates are: 47 percent for females, 42 percent for males, 46 percent for African 
Americans, 39 percent for whites, 48 percent for Hispanics, 49 percent for students who expected 
to complete a four-year college degree, and 34 percent for students who did not. These rates are 
unadjusted. Thus, the difference in completion rates between females and males reflects all the ways 
in which female and male Upward Bound applicants are different, including any differences that may 
exist in, for example, their ages when they enter Upward Bound or their college plans. 

Table II.3 presents adjusted program completion rates that were calculated the same way as the 
adjusted participation rates presented earlier (in Table II.2) to control for differences among 
subgroups. We found a strong association between when a student enters Upward Bound and the 
student’s chances of completing the program. Completion rates rise fairly steadily as entry is 
delayed, with the highest rate obtained for students entering the program after eleventh grade. This 
pattern could have several explanations. The simplest is that students entering late do not have to 
persist very long to complete the program. The estimates suggest that although there may be 
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TABLE 11.3 



ADJUSTED COMPLETION RATES 



Charactemtic 



Cootpi^toii Rate {%> p-valwe for x* 



Sex 



0.42 



Female* 

Male 



47 

42 



Race 

African American* 

Hispanic 

Asian 

White 

Native American 

Entry into Upward Bound 
During 8th grade 
Summer after 8th grade 
During 9th grade 
Summer after 9th grade* 
During 10th grade 
Summer after 10th grade 
During 1 1th grade 
Summer after 1 1th grade 



0.19 

46 

46 

59 

38 

34 

0.03 



25 

36 

43 

49 

50 
62 



Low-income/first generation 

Both low-income and first generation* 
Low-income only 
First generation only 



0.02 

47 

45 

31 



Project director’s rating 
Most likely to serve* 
Somewhat likely to serve 
Least likely to serve 



0.75 

46 

43 

48 



College plans 

Plan to complete a degree* 

Do not plan to complete a degree 



0.01 

49 

33 



Student in racial minority within project 
yes 
no* 



0.88 

45 

44 



Student in sexual minority within project 0.55 

yes 43 

no* 47 . 



Project host institution type and control 




Four-year public institution* 


41 


Four-year private institution 


53 


Two-year institution 


49 





0.14 



TABLE II.3 (continued) 






itSiaraelerfetlc 


Coinpietion Mite (%} 


p-va!ue 


Project is racially diverse (dominant race has less than 60 




0.68 


percent of students) 






yes 


42 




no® 


45 




Project is racially homogeneous (dominant race has 100 




0.33 


percent of students) 


yes 


54 




no* 


43 




Project is sexually diverse (dominant sex has less than 60 




0.34 


percent of students) 


yes 


50 




no* 


44 




Academic requirements 




0.18 


Math/Science* 


45 




Foundational 


36 




Structured 


53 




Unstructured 


36 




Uncategorized 


37 





*The reference attribute. 

SUOl 18 (5/1 1/98, 15:44) and PRED9C.WK4. 
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important benefits to early entry, a higher completion rate is not one of them. As we will see in the 
next section, however, longer duration in the program is a potential benefit." '- 

We also found large differences in completion rates associated with other characteristics: 



• Asian students are more likely to complete the program than white and Native American 
students. Other differences in participation rates by race are not statistically significant, 
even though some are fairly large. 

• Students who are from low-income families are more likely to complete the program 
than students who are not.'^ 

• Students who, before entering the program, expected to earn a college degree are much 
more likely to complete the program than students who did not. 

• Students in “math/science” and “structured” programs are more likely to complete the 
program than other students. However, if it is the high degree of structure that drives 
students to complete the program, it is not clear why students in “foundational” 
programs have relatively low completion rates.'"’ 



"Completion rates are not presented for students entering before ninth grade because those 
students had not graduated before August 1996. The reported rate for students entering during ninth 
grade is based on the subset of those students (about one-fifth of the total) who had graduated before 
that date. The highest possible completion rate for students graduating after August 1996 is 43 
percent. This assumes that all the students still in Upward Bound on that date complete the program. 

'^ot all of the estimated differences are statistically significant. The lowest completion rate— 
for students entering during ninth grade-is significantly different from the rates for students entering 
during the tenth grade or later. The next lowest completion rate— for students entering the summer 
after the ninth grade— is significantly different from the rates for students entering the summers after 
tenth and eleventh grades and, at the 0. 1 1 level, from the rate for students entering during eleventh 
grade. The only other significant difference is between the rates for students entering during the 
summers after tenth and eleventh grades. 

'^The rates for the low-income only and first generation only students are not significantly 
different because these groups are fairly small. In fact, under program regulations, they cannot 
exceed one-third of any project’s students. 



'"’“Math/science” programs require calculus or pre-calculus, at least three science courses, and 
at least six courses in total. “Foundational” programs do not require the math/science curriculum, 
but they do require six courses in total, including reading, writing. Algebra I, Algebra II, and 
geometry. “Structured” programs require six courses in total, but neither the math/science nor the 

(continued...) 
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• Students in projects hosted by four-year private institutions are more likely to complete 
the program than students in projects hosted by four-year public institutions. There is 
no obvious explanation for this. 

We have discovered that many student and project characteristics are associated with differences 
in completion rates. That many are not is equally important. For example, the completion rates for 
males and females are not substantially different. Also, being in the minority (by race or sex) in 
one’s project does not seem to make it either easier or harder to “stick it out” and complete the 
program. Perhaps project staff work hard to retain minority students and successfully address any 
problems such students may encounter. Finally, there is no significant association between the 
project director’s rating of the likelihood of accepting a student and the student’s chances of 
completing the program. Therefore, our inferences about Upward Bound completion rates are not 
affected by our having replaced the usual selection procedures by random assignment; that is, the 
experiences of our study sample reflect the typical experiences of Upward Bound participants. 

C. PROGRAM PERSISTENCE 

The previous section examined the fraction and type of students who complete Upward Bound. 
This section considers how long students are in the program, whether they complete it or not. 
Duration merits examination separate from completion for two reasons. First, noncompleters who 
are in Upward Bound for a long time may derive substantial benefit from the program even though 
they did not finish it. Second, the experience and benefits of a completer who entered the program 



continued) 

foundational curriculum. "Unstructured” programs require fewer than six courses. This program 
typology is described in greater detail in Fasciano and Jacobson (1995). None of the differences 
involving math/science programs is significant, but the differences between structured programs and 
unstructured and foundational programs are significant. 
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in the ninth grade may be very different from the experience and benefits of a completer who entered 
in the eleventh grade. 

Just over one-third of the students entering Upward Bound leave the program within the first 
year, and nearly two-thirds leave within the first two years, as shown in Table 11.4.'^ About one in 
seven remain in the program for three years. The median length of participation in Upward Bound 
is 19 months.'^ ” Among students participating roughly this long (16 to 21 months, in fact), about 
one-third are in Upward Bound for one summer and parts of two academic years. Just over 40 
percent are in for two summers, with such students splitting almost evenly between those in for just 
one academic year and those in for parts of two academic years. 

Students leave Upward Bound either when they complete the program or when they choose to 
leave (or are asked to leave) before completing the program. About 36 percent of the students whom 
we have observed leaving the program left upon completing Upward Bound.'* The remaining 
students left early. We found that the most common reason for leaving Upward Bound before 
completing the program is to take a job (see Table II. 5). That nearly half of the noncompleters drop 
out of the program in May, June, or July suggests, perhaps, that students feel pressure to take a 



'^These estimates of students leaving Upward Bound include students who leave because they 
have completed the program and students who leave early. 

'^These and the other estimates in Table II.4 are from Kaplan-Meier survival functions, which 
properly account for the fact that some students were “censored,” that is, they were still participating 
in the program at the end of the observation period (August 1996). 



”If all students still participating in August 1996 remain in the program until completing it, the 
three year survival rate will end up being about one in six rather than one in seven. However, the 
estimates of the one year and two year survival rates and the median duration will not change. 

'*This figure is lower than'the compietion'rates presented in the previous section because we 
have included in the base of the percentage the program dropouts who had not yet graduated from 
high school. 
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TABLE II.4 



DURATION IN PROGRAM 



Group . 


Percentage in Program for- fl 

Median Duration (months) t year 2 years 3 years 


Overall 


19 


65 


36 


14 


Entry into Upward Bound 










During 8th grade 


19 


67 


42 


42 


Summer after 8th grade 


32 


75 


58 


48 


During 9th grade 


16 


55 


42 


15 


Summer after 9th grade 


24 


70 


49 


15 


During 1 0th grade 


20 


67 


44 


— 


Summer after 10th 


15 


63 


6 


— 


During 1 1 th grade 


17 


59 


— 


— 


Summer after 1 1 th grade 


13 


53 


" 


— 
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TABLE II.5 

MOST COMMON REASONS FOR LEAVING UPWARD BOUND 
BEFORE COMPLETING THE PROGRAM 



: Reason 


Percentage Giving Reason 


Took a job 


30 


Too busy 


12 


Moved 


11 


Asked to leave 


11 
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summer job. This implies that program persistence and completion rates might improve if Upward 
Bound could be made more compatible with work, especially during the summer. 

Not surprisingly, how long a student participates is strongly influenced by when the student 
starts. According to Table II. 4, which presents duration estimates that are not adjusted for student 
and project characteristics, students entering Upward Bound after the eighth or ninth grade persist 
longer than other students. Of course, this effect arises partly because students eventually become 
ineligible. Students entering after eleventh grade, for example, can only be in the program for 12 
to 14 months, depending on whether they attend a summer bridge program.” Students entering at 
the end of eighth grade have the opportunity to remain in the program much longer. For some key 
subgroups defined by characteristics other than time of entry into the program, unadjusted median 
durations are-in months-20 for females, 1 7 for males, 20 for Afncan Americans and whites, 1 5 for 
Hispanics, 21 for students who expected to complete a four-year college degree, and 15 for students 
who did not. 

In Table II. 6, we present median durations and 12- and 18- month survival rates.*® These 
estimates are adjusted for a wide range of student and project characteristics, like the participation 
and completion rates presented earlier. 

According to Table II.6, the largest differences in persistence are associated with when a student 
entered the program. As noted before, students entering Upward Bound before the tenth grade tend 



”A student who does not graduate from high school on time could participate longer. 



^°The median duration is the duration at which the percentage remaining in the program drops 
below 50 percent. The 12 and 1 8 month survival rates are the estimated percentages of participants 
remaining in the program for at least 12 and 18 months, respectively. Estimates were calculated 
from a discrete-time proportional hazards model. 
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TABLE II.6 



ADJUSTED DURATION ESTIMATES 



. Characteristic 


Mediae Deratioa 


^ So rviva): Rate (%) 

12 nioBth 1 8 Month : 


p-valuefor x* 


Sex 

Female* 


20 


65 


52 


0.35 


Male 


19 


63 


50 




Race 

African American* 


20 


63 


52 


0.26 


Hispanic 


22 


68 


58 




Asian 


24 


70 


61 




White 


22 


67 


57 




Native American 


22 


68 


58 




Entry into Upward Bound 
During 8th grade 


25 


70 


58 


0.00 


Summer after 8th grade 


35 


82 


74 




During 9th grade' 


16 


60 


46 




Summer after 9th grade* 


24 


69 


56 




During 10th grade 


13 


53 


37 




Summer after 10th grade 


II 


43 


27 




During 1 1th grade 


6 


29 


15 




Summer after 1 1th grade 


5 


22 


10 




Low-income/first generation 

Both low-income and first generation* 


19 


65 


51 


0.45 


Low-income only 


22 


69 


56 




First generation only 


19 


64 


51 




Project director’s rating 
Most likely to serve* 


20 


65 


53 


0.94 


Somewhat likely to serve 


20 


65 


53 




Least likely to serve 


21 


66 


54 




College plans 

Plan to complete a degree* 


21 


65 


54 


0.19 


Do not plan to complete a degree 


18 


61 


50 




Student in racial minority within project 
yes 


18 


64 


50 


0.46 


no* 


20 


66 


53 




Student in sexual minority within project 
yes 


17 


61 


48 


0.04 


no* 


20 


66 


53 




Project host institution type and control 
Four-year public institution* 


16 


59 


46 


0.03 


Four-year private institution 


22 


66 


55 




Two-year institution 


21 


65 


53 
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TABLE 11.6 {continued) 



Characteristic 


Mediao Duration 


Survival Rate (%) 

12 nioRtli 18 Month 


■Rvalue for 


Project is racially diverse (dominant race has 








0.47 


less than 60 percent of students) 


yes 


17 


60 


48 




no* 

Project is racially homogeneous (dominant 


18 


62 


50 


0.31 


race has 1 00 percent of students) 


yes 


20 


65 


52 




no* 

Project is sexually diverse (dominant sex has 


17 


62 


48 


0.44 


less than 60 percent of students) 


yes 


18 


62 


49 




no* 

Academic requirements 


16 


60 


46 


0.48 


Math/Science* 


16 


56 


45 




Foundational 


15 


53 


42 




Structured 


17 


59 


48 




Unstructured 


19 


61 


50 




Uncategorized 


15 


54 


43 





*The reference attribute. 

SUOl 18 (5/1 1/98, 15:44) and PRED9B.WK4. 
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to remain in the program longer than students entering later. Students entering during the summer 
after eighth grade persist the longest at a rate significantly different from the rate for any other group. 
The shortest durations are observed for students entering Upward Bound during or after the eleventh 
grade, with all observed differences in rates being significant. 

Other characteristics that are associated with persistence in Upward Bound are: race, whether 
the student is in the sexual minority within a project, and type and control of the project's host 
institution. Native American, Asian, Hispanic, and white students have similar patterns of program 
persistence, and they remain in the program longer than African American students, although the 
Native American-African American difference is not statistically significant. Large and statistically 
significant differences are also associated with the type and control of the host institution. For 
reasons that are not clear, the lowest persistence rates are found for students whose Upward Bound 
programs are hosted by public four-year institutions, rather than private four-year institutions or two- 
year institutions. 

D. PROGRAM INTENSITY 

The preceding sections in this chapter examined whether students participate in Upward Bound, 
whether they complete the program, and how long they are in it. This sections looks at what students 
do while they are in Upward Bound. 

Upward Bound is an intensive program in which participants engage in many academic and 
nonacademic activities. Table II.7 shows that during a typical Upward Bound summer program, the 
average student attends just over 100 sessions of academic courses and about 70 sessions of 




54 



32 



TABLE II.7 



STUDENT PARTICIPATION IN UPWARD BOUND ACADEMIC COURSES 
AND NONACADEMIC ACTIVITIES DURING THE SUMMER 



Typical Number of Sessions Attended* ■ 

^ Percentage Enrolled ' All Students* . . Enrolled Students* 



All courses: 


99 


103 


104 


English 


88 


30 


30 


ESL 


0 


0 


24 


Foreign languages 


22 


0 


23 


Math 


95 


24 


24 


Computers 


24 


0 


24 


Science 


72 


20 


24 


Social science 


14 


0 


24 


Elective 


77 


20 


24 


Other 


6 


0 


24 


All activities: 


100 


72 


72 


College preparation 


93 


13 


14 


Career exploration 


45 


0 


3 


Self-awareness 


58 


1 


2 


Field trips 


91 


4 


4 


Cultural awareness 


58 


1 


2 


Counseling 


92 


13 


17 


Skill development 


87 


21 


24 


Other 


6 


0 


1 


All courses and activities 


100 


181 


I8I 



“"Typical" refers to the median number of sessions attended and number of sessions is defined as the number 
of times a student attended a course, for example. 

**“ All students” refers to all Upward Bound participants, regardless of whether they actually enrolled in a 
course. “Enrolled students" refers to only those who were actually enrolled. 
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BEST COPY AVAfUBLE 



nonacademic activities.-' Nearly all students take at least one math course, and almost 90 percent 
take at least one English course. Nearly three-quarters of the students take a science course and an 
elective. Other types of courses are taken by relatively few students. Students taking a course in a 
particular subject attend about 24 sessions, except in English for which the average number of 
sessions attended is 30. These figures imply about 3 to 5 meetings per week for an Upward Bound 
summer course. 

During the summer, participation rates are high-sometimes exceeding 90 percent— for most of 
the nonacademic activities listed in Table II. 7. The activities involving the most sessions are college 
preparation, counseling, and skill development, with 14 to 24 sessions-2 to 3 per week-attended 
by students engaged in those activities. 

Not surprisingly, students are much less active in Upward Bound during the academic year. 
According to Table II. 8, the average student attends 36 sessions of academic courses and 43 sessions 
of nonacademic activities. About one-quarter of the students take no academic courses through the 
Upward Bound program. Participation rates in science, math, and English range from one-half to 
two-thirds. Students taking a course in one of those subjects typically attend 1 5 to 20 sessions 
during the academic year. Although participation rates for most nonacademic activities are high 
during the academic year, attendance tends to be infrequent. 



^'The estimates in Table II. 7 pertain to the first summer of program participation by students 
who participated at least one summer. These figures reflect a typical summer experience because 
we find no evidence suggesting that a student’s level of activity rises or falls substantially over time 
so long as the student remains in the program. Two related findings are (1) the level of activity does 
not vary much according to the student’s grade in school and (2) students who remain in the program 
only a short time appear to participate as fully while they are in the program as the seemingly more 
dedicated students who are in the program much longer. We obtain similar findings for the academic 
year. Hence, the estimates presented in Table II. 8 later in this section pertain to the first academic 
year of program participation by students who participated at least one academic year. 
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TABLE 11.8 



STUDENT PARTICIPATION IN UPWARD BOUND ACADEMIC COURSES 
AND NONACADEMIC ACTIVITIES DURING THE ACADEMIC YEAR 



Typical Number of Sessions Attended* 


Percentage Enrolled 


All Student 


i Enrolled Students'* 


All courses: 


75 


36 


60 


English 


67 


12 


20 


ESL 


0 


0 


10 


Foreign languages 


7 


0 


14 


Math 


64 


8 


15 


Computers 


7 


0 


17 


Science 


50 


0 


15 


Social science 


7 


0 


15 


Elective 


42 


0 


15 


Other 


2 


0 


6 


All activities: 


99 


43 


45 


College preparation 


81 


5 


9 


Career exploration 


20 


0 


2 


Self-awareness 


61 


1 


3 


Field trips 


73 


2 


3 


Cultural awareness 


58 


I 


2 


Counseling 


92 


11 


12 


Skill development 


87 


12 


15 


Other 


4 


0 


1 


All courses and activities 


100 


81 


81 



“"Typical” refers to the median number of sessions attended. 

*’"AI1 students” refers to all Upward Bound participants regardless of whether they actually enrolled in 
a course. "Enrolled students” refers to only those who were actually enrolled. 
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As noted earlier, the total number of academic or nonacademic activities in which a student is 
engaged during a summer or an academic year does not seem to rise or fall as the student remains 
in the program longer and does not seem to vary according to the student's grade in school. 
Furthermore, all of the students in a given Upward Bound project are attending roughly the same 
number of academic and nonacademic sessions. In other words, most of the observed differences 
in activity levels among students are attributable to differences among projects in program offerings 
and staff expectations.’- However, after examining relationships between average attendance figures 
and various project characteristics, we found no consistent evidence suggesting that students in some 
types of projects are more active than students in other types of projects. 

The measures of intensity examined so far pertain to periods of well-defined and limited 
duration— a summer or an academic year. For a final measure of intensity, we will consider students’ 
entire Upward Bound careers, which can span just part of one summer or academic year or several 
summers and academic years. Whereas, earlier, we measured the dose of Upward Bound received 
by a student by the number of months the student participated, we will measure the dose here by the 
number of activities in which the student participated. We find that the typical student attends 
roughly 375 Upward Bound sessions, about 55 percent of which are academic courses and the 
remainder are nonacademic activities.^^ Over the course of an average student’s Upward Bound 



Across different measures of activity levels (numbers of sessions attended), we found that 65 
to 85 percent of the variability among all students is attributable to variability among students in 
different projects, and only 15 to 35 percent is attributable to variability among students in the same 
project. Thus, while the activity levels of different students in a given project vary only narrowly 
about the project average, the average activity levels in different projects vary widely about the 
average for the whole Upward Bound program. 



^^This total career sessions figure surely understates somewhat the typical dose of Upward 
Bound because as noted earlier, about 13 percent of the students in the evaluation sample were still 
active and had up to one more year in the program when their activities were last reported to us. 
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career, about 30 percent of the academic sessions attended are in English, 20 percent are in math, 
and 15 percent are in science. Thus, there is a strong focus on core academic subjects. 
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III. THE IMPACT OF UPWARD BOUND 



Increasing the skills and motivation of students to prepare them to succeed in college is an 
important goal of the Upward Bound program. This evaluation examines many outcomes related 
to this goal, including students’ expectations concerning how far they will go in school, the range 
of courses they completed in high school, and their performance in these courses. Furthermore, 
since the students in our study sample who entered Upward Bound in 1 0th or 1 1 th grade had an 
opportunity to attend college for one or two years, we can begin to examine the program’s impact 
on the college experience. 

In this chapter, we examine Upward Bound’s impact on student outcomes, such as educational 
expectations, high school courses, grades, misbehavior in school, parent involvement, college 
enrollment and selectivity of the college attended, access to financial aid, credits earned in college 
courses, and participation in college activities. Specifically, we address the following questions: 

• What is the impact of Upward bound on all students who were randomly selected for the 
program and what are the effects of participating in Upward Bound on students’ 
outcomes? 

• Do some groups of students benefit from Upward Bound more than others? 



Our findings include the following: 




• Upward Bound has limited impacts on students during high school. When we looked 
across a variety of outcomes, we found that Upward Bound had impacts on only a few. 
Roughly two to three years after being selected for Upward Bound, students in the 
treatment group expected to complete slightly more years of education and had earned 
slightly more credits in math and social studies than students in the control group. 
Upward Bound generally had no impacts on students’ in-schools behavior, participation 
in extra curricular activities, grade point average, or credits earned in subjects such as 
English or science. Furthermore, Upward Bound had no impact on high school 
graduation. 

L GO 



• Upward Bound may have some impact on participants* postsecondary education 
experiences. Although, Upward Bound had no impact on the chances students attended 
college, students in the treatment group earned more credits from four-year colleges, 
were more likely to receive financial aid, and were more actively engaged in some 
college activities. But since relatively few study participants had reached college age 
and those who did entered Upward Bound during grades 10-12, it is too soon to make 
definitive statements about Upward Bound’s impact on postsecondary outcomes. 

• Upward Bound has substantial impacts on some groups of students and not others. 
Although Upward Bound had small impacts on students as a whole, some groups of 
students received greater benefits. Focusing on the key outcomes of educational 
expectations and courses taken in high school, we found that (1) students with lower 
initial educational expectations benefitted substantially more than those with higher 
expectations, (2) boys showed substantially larger impacts than girls, (3) Hispanic and 
white students benefitted more than African American students, (4) students who were 
low-income only or low-income and potential first-generation college students showed 
larger impacts than those who qualified for the program only as potential first- 
generation students, and (5) poorer performing students benefitted substantially more 
than their better performing peers. 



As part of this evaluation, we also explored whether there was an association between project- 
level characteristics and project-level impacts. This analysis showed that there were only small 
differences in impacts among projects.' Furthermore, no consistent relationships existed between 
project characteristics, such as number of students served, composition of the student population, 
or average duration in Upward bound by project participants and the size of project impacts. 

This chapter describes our method of measuring the impact of Upward Bound on students’ 
outcomes. Next, we present a context for interpreting the findings. Finally, we describe impact of 
the program for all students and some subgroups of students. 



'Generally, less than five percent of the total variation in student outcomes could be attributed 
to differences among project impacts. 
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A. COMPUTING PROGRAM IMPACTS 



An important feature of the evaluation was the application of random assignment to construct 
two statistically equivalent groups of students: (1) a treatment group and (2) a control group. 
Because students were randomly selected for the treatment group and the control group, the only 
difference between them should be the offer of Upward Bound services to the treatment group and 
those students’ participation in the program. The difference between the two groups on a given 
outcome— for example, the average number of credits earned in high school— provides an estimate 
of the program’s impact. 

Two sets of program impacts are presented in this chapter. One set of estimates shows the 
impact of being offered the opportunity to participate in the Upward Bound program; that is, the 
impact of Upward Bound on all students randomly selected to participate in Upward Bound 
(assigned to the treatment group). These impact estimates combine the effects of two processes: 
(1) students showing up (or not showing up) for Upward Bound services and (2) students actually 
participating in the program. By focusing the analyses on the impact of students being given the 
opportunity to participate in Upward Bound, we define the treatment more broadly than simply what 
the Upward Bound staff accomplishes once students participate in various Upward Bound activities 
and courses; it also includes the projects’ efforts in getting students to show up for services and their 
exposure to these services. About 1 8 percent of the students who were eligible for the program did 
not participate at all and about 65 percent of all those selected for Upward Bound had stopped 
participating before the end of 12th grade. Furthermore, many students who showed up for services 
decided to leave the program rather early on— almost 40 percent left within the first 12 months. 

The second set of estimates we present show the impact of actually participating in Upward 
Bound', these estimates adjust for students who never showed up for services and indicate the impacts 
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for those who attended at least one session.- For this study, we assumed that ( 1 ) the same proportion 
of students in the control group and the treatment group, when given the opportunity to participate 
in Upward Bound, would decide to participate and (2) Upward Bound has no impact on students 
who never attended a course or activity.^ Since we had to make these untestable assumptions when 
computing impacts for program participants and these estimates only tell part of the story concerning 
the program’s overall impact on students, we focus our discussion on the findings pertaining to all 
students who were offered the opportunity to participate in Upward Bound; however, we present 
both sets of estimates in tables so that readers may compare findings for all students and for 
participants. In general, the conclusions drawn from the two sets of estimates are similar. 

B. CONTEXT FOR INTERPRETING PROGRAM IMPACTS 

When interpreting the findings concerning program impacts, it is useful to keep two contextual 
features in mind: ( 1 ) many students in the study were too young to have attended college by the time 
we last collected information in 1996, and (2) students in both the treatment and control groups 
received other services. 



^The estimates, however, do not take account of the fact that some students only participate for 
a few months while other students participate for three or four years. Chapter IV examines the effect 
of diiration in Upward Bound on students’ outcomes. 



^Generally, impacts adjusted for no-shows are larger than those we compute for all students; 
however, because we used different analytic approaches for computing program impacts for all 
students and for participants we may find impacts for participants that are smaller than for all 
students. When computing the impact for all students we used a procedure referred to as 
subclassification analysis and when adjusting for no-shows we used an analytic model to compute 
program impacts. A complete discussion of these approaches is described in Appendix 2.A. 
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1. Many Students in the Study Are Too Young to Have Attended College 



Most students who applied for Upward Bound in late 1992 through early 1994 were in 9th or 
10th grade when they applied; a few were in 8th grade or 11th grade. This has important 
implications when examining students’ 
college enrollment. When we last collected 
information from students, many were still 
in high school (see Figure IILl) and Dropped out of high school 
findings concerning college enrollment 



FIGURE 111.1 

SCHOOL STATUS IN 1996 



Attended College 



reflect only the experiences of about one- 
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quarter of all students in the study. 

Moreover, these findings relate to the experiences of students who entered Upward Bound later in 
high school and had comparatively few years of exposure to the program. A more comprehensive 
picture of the experiences of all students in the study will have to wait until after late 1998 when we 
again collect data from the students. 



2. Estimates of Program Impact Show Upward Bound’s Value Added 

The experiences of students in the control group show the educational outcomes we would 
expect in the absence of the Upward Bound program. These outcomes are the product of many 
factors, such as high school attendance, parents’ efforts to motivate students and help them with their 
homework, peer interaction, tutoring, counseling, and other pre-college programs. The results from 
the most recent data collection show that almost half the students who were in the control group 
received some kind of supplemental service between the time they applied and 1996: almost 60 
percent of all students reported receiving services during at least one school year and more than 20 
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percent reported participating in a program during at least one summer (see Table III. I)/ These 
students received a broad range of services that included tutoring and counseling, and some 
participated in more formal programs such as the TRIQ Talent Search program.' Upward Bound 
students also reported receiving supplemental services; however, they generally reported 
participating less in these services than members of our control group. 

Since we compute the impact of Upward Bound by comparing the outcomes of the Upward 
Bound group with the control group, the impacts show how much the program leads to changes 
beyond the experiences of similar students who were not in Upward Bound.* This difference is the 
value added by Upward Bound over these outside factors. If Upward Bound is only as effective as 
the services that students already have available, we would expect the Upward Bound students and 
the students in the control group to have similar outcomes.^ 



'‘About 41 percent of students in the treatment group reported that they too had participated in 
supplemental service programs other than Upward Bound. Some Upward Bound students may have 
participated in these programs after leaving Upward Bound. 

*Up to 1 1 percent of the control group, for example, participated in programs referred to as 
Talent Search/Educational Talent Search. Because students often were not specific when they 
reported this information, it is impossible to identify all students who participated in the TRIO 
funded program. 

*A few students selected for the control group were given Upward Bound services by projects 
that participated in the evaluation. Also, a few students obtained Upward Bound services from 
Upward Bound projects that were not in the evaluation. To maintain the integrity of the random 
assignment design, we treated all students selected for the control group as controls, regardless of 
whether they participated in Upward Bound. 



^Appendix D shows the extent to which various subgroups of students participated in 
supplemental services. 
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TABLE III.l 



USE OF SUPPLEMENTAL SERVICES OTHER THAN 
THOSE PROVIDED BY UPWARD BOUND 





Upward i 
Bound 


Group 


Any Supplemental Service 


Ever 


41% 


58% 


Summer 


15 


21 


Academic Year 


39 


54 


Program Attended Most 


Regular Upward Bound (other than project applied to) 


0 


1 


Upward Bound Math/Science 


12 


13 


Talent Search 


7 


11 


Other pre-college 


2 


4 


Tutoring 


28 


41 
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C. PROGRAM IMPACTS 

This section describes Upward Bound’s impacts on students’ outcomes. These findings apply 
to the eligible students who applied for Upward Bound during 1992-94. A broad range of outcomes 
were analyzed, such as college attendance, postsecondary credits earned, receipt of financial aid, 
high school completion, high school course taking, participation in high school activities, school- 
related behavior, and parent involvement. 
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1. High School Graduation, Educational Expectations, and High School Course Taking 

a. Upward Bound Had No Impact on High School Graduation Status 

When we last collected data, there was a small, but statistically insignificant difference in the 
percent of Upward Bound students who had graduated from high school and the percent of students 
in the control group who had graduated; 59 percent of the Upward Bound students had graduated 
from high school and 63 percent of those in the control group had graduated. While Upward Bound 
had no impact on high school graduation, it did appear that relatively more Upward Bound students 
were still in high school when we last collected data than students in the control group (35 percent 
of the treatment group were in high school and 28 percent of the control group were in high school). 

b. Being Selected for Upward Bound Led to Higher Educational Expectations 

Each time we collected information from students, we asked them about their educational 
expectations-how many years of schooling they think they would complete and how far in school 
their parents think they will go. Upward Bound students reported that they expected to complete 
about 0.3 more years of education than similar students not selected for the program (see Table 
III. 2). On average, students in Upward Bound reported that they expected to complete 16.4 years 
of schooling. On the other hand, those in the control group said they expected to complete 16.1 
years. Further analysis suggests that this difference in expectations arises in part because more 
students in Upward Bound expected to complete post-baccalaureate work than those in the control 
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TABLE III.2 



IMPACT OF UPWARD BOUND ON STUDENTS' EXPECTATIONS 
AND HIGH SCHOOL COURSE TAKING 





All Students 




Participants ' 




Treatment 

Mean 


Control 

Mean 


Impact 


V Impact 


Educational Expectations (years of schooling) 


Students’ 


16.4 


16.1 


0.3* 


0.4* 


Fathers’ 


16.9 


16.8 


0.0 


0.1 


Mothers’ 


17.1 


17.0 


0.1 


0.1 


High School Credits” 


Non-remedial English 


3.8 


3.7 


0.1 


0.1 


Non-remedial social studies 


2.8 


2.6 


0.1* 


0.2 


Non-remedial math 


2.8 


2.6 


0.2* 


0.3* 


Non-remedial science 


2.6 


2.5 


0.1 


0.1 


Non-remedial foreign language 


1.4 


1.4 


0.0 


0.0 


Non-remedial total for 5 major subjects 


13.5 


12.9 


0.6 


0.6 ' 


Non-remedial vocational education 


1.6 


1.7 


-0.1 


-0.1 


Non-remedial computer science 


0.8 


0.9 


-0.0 


0.0 


Non-remedial other 


4.1 


4.0 


0.1 


0.1 


Total non-remedial 


20.0 


19.5 


0.6 


0.7 


Total AP/honors, all subjects 


1.9 


2.0 


-0.1 


-0.0 


Total credits, includes remedial 


20.4 


19.9 


0.6 


0.7 


Satisfied New Basics Curriculum 


30% 


29% 


1“ 


1 


Cumulative CPA 


2.3 


2.3 


0.0 


0.0 


School Status 


Still in High School 


35% 


28% 


7^b 


8* 


Dropped out 


6 


9 


-3 


-4 


Graduated 


59 


63 


-3 


-4 



♦Significant at the .10 level using a one-tailed t-test. 

“High school credits are defined in terms of Carnegie units. A Carnegie unit corresponds to a course that meets 
for 45-60 minutes, five days a week, for an entire academic year. 

•’Impact expressed as percentage points. 
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group (see Figure III.2). We found 



FIGURE III.2 



no impacts for parents’ 
expectations concerning years of 
schooling, as reported by the 
students. We found no impacts for 
parents’ expectations concerning 
years of schooling, as reported by 
the students. 



IMPACT OF UPWARD BOUND ON STUDENTS’ 
EDUCATIONAL EXPECTATIONS 



High school or less 



Some college 



Four-year college degree 



More than four-year college degree 



Upward Bound 
O Control Group 




Percent 



c. Upward Bound Had Small Impacts on Credits Earned in High School 

Using data from students’ high school transcripts, we found that among all students who were 
selected for Upward Bound, the program had only small impacts on the number of credits students 
had earned (see Table III.2).* Upward Bound students earned about 0.2 more credits in math than 
students in the control group and about 0. 1 more credits in social studies by the time most of them 
had reached their senior year in high school or graduated. These impacts correspond to about an 8 
percent increase in math credits and about a 5 percent increase in social studies credits over what 
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*The analyses described by Myers and Schirm (1997), where larger impacts were found, focused 
on the course taking of students who began participating in Upward Bound as a high school 
sophomore or later and examined gains in credits earned between their freshman year and the end 
of the 1993/94 school year. This approach was used to adjust for possible differences in the initial 
number of credits earned for treatments and controls. A different approach was used here (see 
Appendix B) and all students were included in the analyses. We employed this new approach 
because many of the outcomes used in this analysis did not have baseline measures. For example, 
without limiting the sample, high school credits could not be measured before students entered 
Upward Bound. Also, by using this approach we were able to include all students in the analysis 
of high school credits and not just those who entered Upward Bound in grade 10 or later. In part, 
the larger impacts found in the earlier report may have reflected differences in approaches or because 
we only took a snapshot of the students’ experiences after they had only completed a few years of 
high school. 
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students would have earned in the absence of Upward Bound. We found positive, but statistically 
insignificant impacts for the other core subject areas— English, science, and foreign language. 

In the past decade, some states have initiated tougher standards concerning the core curriculum 
that students must complete to obtain a high school diploma; often this curriculum is referred to as 
the New Basics (A Nation at Risk 1983). The requirements of the New Basics Curriculum include 
earning four credits in English, three credits in mathematics, three credits in science, and three 
credits in social studies.’ Although many Upward Bound students completed the core curriculum, 
we found no significant difference between the treatment and control groups: 30 percent of all 
Upward Bound students had completed the core curriculum and 29 percent of the control group had 
completed it (see Table III.2). 

To compute the impact of Upward Bound, we use all students in the treatment and control 
groups regardless of whether or not they had completed high school. As a consequence, the 
percentage of Upward Bound students and students in the control group who completed the New 
Basics Curriculum is lower than would be expected if we Just looked at high school graduates. If 
we compare the experiences of high school graduates in Upward Bound and the control group, we 
find that about 44 percent of the Upward Bound high school graduates and the control group 
graduates had completed the curriculum. 



’The authors of a Nation At Risk also included one-half credit of computer science. NCES does 
not include this last element in its deftnition of the New Basics curriculum and we have followed 
the practice in this report. Nationwide, about 50 percent of all 1994 high school graduates had 
completed the New Basics curriculum (U.S. Department of Education, 1 996) and 44 percent of the 
Upward Bound high school graduates had completed this core curriculum. The most common 
subject requirement that these students had not completed was earning three credits in science and 
20 percent of the students had not completed any of the requirements. (Source: UBSC2201.) 
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d. Upward Bound Had No Impact on Extra Curricular Activities or Parent Involvement 

We found no impacts for a wide range of the outcomes dealing with participation in 
extracurricular and community activities, school misbehavior and parent involvement; and for some 
outcomes we found that students in the control group had better outcomes than Upward Bound 
students.'® For example, students who were selected for Upward Bound were just as likely as those 
in the control group to report that they were late for school, skipped classes, or had been in trouble 
at school for not following rules (see Table III.3). Students in Upward Bound were less likely than 
students in the control group to participate in high school activities such as student government, the 
school newspaper, or yearbook. Finally, Upward Bound students reported that they were less likely 
to participate in church or community groups. Limited participation in these activities should not 
be particularly surprising given the time commitment that Upward Bound projects often demand of 
students. During a typical summer and academic year program, students attend about 250 sessions 
concerning formal instruction in academic subjects, tutoring, and counseling. 

2. Going to College 

The most important goal for Upward Bound is increasing students’ access to college. We used 
two approaches to collect data on college access and related outcomes. First, we conducted a survey 
of students and asked them about college attendance. Second, we collected students’ transcripts 



'®In almost all cases we used a one-tailed test of statistical significance to assess the chances of 
a positive impact being as large or larger than that observed in the sample, by chance alone. For 
some analyses where the direction of the expected impact was unclear, we used a two-tailed test. 
For example, analysis of impacts of participating in extracurricular activities relied on two-tailed 
tests. 



TABLE III.3 



IMPACT OF UPWARD BOUND ON STUDENTS' HIGH SCHOOL EXPERIENCES 







All Stadents 




Participants 




Treatmeot Control 
Mean Mean 


Impact 


Impact 


Participated in Activities in High School (Percent Yes): 


Band, orchestra 


25% 


29% 


-4 


-5%# 


Drama club, school play 


16 


22 


-6# 


-6 


Student government 


17 


23 


-6# 


-7 


Academic honor society 


21 . 


25 


-4 


-4 


School newspaper or yearbook 


18 


25 


-7# 


-9# 


Service club 


20 


25 


-5# 


-6 


Academic club 


41 


42 


-I 


0 


Hobby club 


12 


10 


2 


3*# 


Vocational education club 


30 


37 


-7# 


-8 # 


Team sports 


26 


26 


0 


-I 


Individual sports 


17 


17 


-1 


-2 


Cheerleading, pom pom 


II 


II 


0 


0 


Church activities, youth groups 


37 


46 


-10# 


-12 


Community groups 


10 


15 


-5# 


-6# 


Misbehavior (Number of times): 


Late for school 


4.4 


4.3 


O.I 


O.I 


Skipped classes 


2.0 


1.8 


0.2 


0.2 


Missed a day of school 


5.0 


4.9 


O.I 


O.I 


In trouble for not following school rules 


0.9 


0.9 


0.0 


0.0 


Put on in-school suspension 


0.3 


0.4 


-O.I 


-O.I 


Suspended 


0.2 


0.2 


-0.0 


-O.I 


Transferred to another school 


0.0 


0.0 


-0.0 


-0.0 


Arrested 


O.I 


O.I 


-0.0 


-0.0 


Spent time in juvenile home 


0.0 


0.0 


-0.0 


-0.0 


Tried and convicted as an adult 


0.0 


0.0 


0.0 


0.0 


Discuss with Parent Sometimes/Often (Percent Yes): 


About selecting courses 


67% 


73% 


-6%# 


-8% 


About school activities 


72 


76 


-4 # 


-5 


About studies 


65 


75 


-10 # 


-12 


About grades 


84 


87 


-3 # 


-3 


About transferring to another school 


14 


15 


-I 


-2 


About taking, preparing for ACT/SAT 


67 


68 


-I 


-2 


About going to college 


89 


92 


-3 


-4 
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TABLE III.3 (continued) 





Ail StBdents 
Treatmeot;:?: :: 

Meao ' Mmq 


Impact 


Participants 

Impact 


Parent Sometimes/Oflen (Percent Yes): 


Checked on homework 


64% 


63% 


1% 


1% 


Helped with homework 


48 


48 


0 


-0 


Gave special privileges for good grades 


61 


61 


1 


1 


Limited privileges because of poor grades 


44 


42 


1 


2 


Required chores around the house 


84 


87 


-3 


-3 


Limited time watching TV, playing video games 


37 


34 


3 


4* 


Limited time with friends 


59 


57 


2 


2 


Did parent... (Percent Yes): 


Attend a school meeting? 


47% 


57% 


-10%# 


-12%# 


Speak with teachers? 


69 


71 


-2 


-3 


Visit classes? 


37 


43 


-7 # 


-7 # 


Attend a school event? 


53 


62 


-8 


-11 



‘Significant at the .10 level using a one-tailed t-test. 
#Signiflcant at the .10 level using a two-tailed test. 
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from all postsecondary institutions they told us about in the student survey. Using both types of data 
we compiled a picture of students’ college attendance, credits earned in college, type of school 
attended (four-year, two-year, or vocational), selectivity of the college attended, engagement in 
college life, and receipt of financial aid. 

While we must be cautious not to place too much emphasis on the findings concerning college 
outcomes because our data only reflect the experiences of students who entered Upward Bound later 
in high school, the results show that in the short-term. Upward Bound students were no more likely 
than control group students to have ever attended college (either a two- or four-year college), and 
the four-year colleges they attended were no more selective than those attended by the control 
group.” Upward Bound students, however, earned more postsecondary credits than students in the 
control group, particularly in four-year colleges.'- Students in Upward Bound, were also more likely 
to receive financial aid and were often more involved in college activities. 

a. Students Selected for Upward Bound Earn More Postsecondary Credits 

Although students in Upward Bound and the control group were equally likely to have ever 
attended a postsecondary school. Upward Bound students earned more non-remedial postsecondary 
credits (see Table III.4). Students who were selected for Upward Bound earned about 



"Selectivity of the college was based on Peterson’s 1998 Four- Year college Selectivity Index. 
Scores ranged from 1 (non-competitive) to 5 (most difficult). Scores were based on the percentage 
of 1996 applicants who were, accepted, and, oh the high school rank and standardized test scores of 
the accepted freshman. Two-year colleges and vocational schools were given a score of zero. 
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TABLE III.4 



IMPACT OF UPWARD BOUND ON COLLEGE ENROLLMENT 
AND RELATED OUTCOMES 







All Students 




Participants 




Treatment 

Mean 


Control 

Mean 


Impact 


Impact 


School Status Indicator 


Attend college 


23% 


25% 


-3“ 


-2 


Attend four-year college 


15 


16 


-1 


0 


Attend two-year college 


5 


8 


-3 


-4 


Attend vocational school 


2 


1 


1 


1 


Postsecondary Credits'" 
Any Type School 


Total non-remedial 


6.8 


5.7 


1.1* 


1.7* 


Total remedial 


0.2 


0.4 


-0.3* 


-0.4* 


Four-Year College 


Total non-remedial 


5.7 


4.4 


1.3* 


2.0 


Total remedial 


0.1 


0.1 


1 

o 

b 


-0.0 


Two-Year College 


Total non-remedial 


1.0 


1.2 


-0.2 


-0.3 


Total remedial 


0.0 


0.2 


-0.2* 


-0.2* 


Vocational/Technical School 


Total non-remedial 


0.2 


0.1 


0.0 


0.0 


Total remedial 


NA 


NA 


NA 


NA 


College Selectivity 


0.8 


0.9 


-0.12 


-0.03 



•Significant at the .10 level using a one-tailed t-test. 

NA = unable to estimate. 

“Impacts on outcomes concerning school status expressed as percentage points. 
'’Postsecondary credits are defined in terms of semester hours. 
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seven credits and students in the control group earned about six credits.'^ When we compared 
students’ experiences earning credits in remedial courses, we found that Upward Bound students 
earned slightly fewer remedial credits than students in the control group (0.3 credits less). 

Separating the findings for each of these school types showed that Upward Bound led to an 
increase of 1.3 credits in the number of non-remedial credits earned in four-year colleges and to 
reduction of 0.2 credits earned in remedial courses while attending two-year colleges. This suggests 
that participation in Upward Bound helps to better prepare students for college course work. 
Upward Bound did not have a substantial impact on credits earned in vocational schools. 

b. Upward Bound Students are More Likely to Receive Financial Aid 

We found that the Upward Bound group had a somewhat greater chance of receiving financial 
aid than the control group: 33 percent of those in Upward Bound report receiving aid. compared to 
30 percent of the control group (see Table III.5).''* The most common form of financial aid for both 
groups was a Pell grant (24 percent of the treatment group and 2 1 percent of the control group 
received a Pell grant). The next most common form of aid was a loan, and Upward Bound students 
were more likely to receive loans as well. Compared to students in the control group, students in 



'^The values for the average credits earned while attending a postsecondary institution are lower 
than one might expect because they refer to all students selected for Upward Bound and the control 
group, and not just those who attended college. We cannot make a direct comparison of college 
goers in the treatment group and the control group to compute the program’s impact on those who 
attended college because the two groups will be no longer statistically equivalent and biased 
estimates of project impacts will be computed. To provide some perspective, however, we note that 
among those who attended a four-year college. Upward Bound students earned 34 credits and 
students in the control group earned 27 credits. 



'^Almost all students in the study who attended college received some type of financial aid. 
This should not be surprising given that more than 80 percent of the Upward Bound applicants were 
from low-income families. 
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TABLE III.5 



IMPACT OF UPWARD BOUND ON STUDENTS^ RECEIPT OF FINANCIAL 
AID AND SOURCES OF INFORMATION CONCERNING FINANCIAL AID 



All Students : ■ Participants 

Treatment CoBirol 

Mean Mean Impact Impact 

33% 30% 3*#* 



Received any type of financial aid 
Received loans 
Received tuition waiver 
Received Pell Grant 

Received another type of grant or fellowship 
Received a work/study appointment 

Did to learn about financial aid 
Talk with teacher/counselor 
Talk with college representative 
Talk with loan officer at a bank 
Read Dept, of Education information 
Read information from a college 
Read about aid available through military 
Talk to a knowledgeable adult 
Talk with friends 



22 


20 


3* 


3* 


8 


7 


1 


1 


24 


21 


3*U 


3 


19 


16 


3*U 


3*U 


11 


7 


4*U 


4*U 



78% 


81% 


-2 


-3 


61 


62 


-1 


-2 


3 


6 


-3U 


-3 U 


40 


42 


-2 


-2 


73 


67 


5*U 


6*U 


35 


43 


-SU 


-10 


83 


80 


3* 


3 


75 


76 


-1 


-3 



■^Significant at the .10 level using a one-tailed t-test. 
^Significant at the .10 level using a two-tailed t-test. 
“Impacts expressed as percentage points. 



Upward Bound were also more likely to obtain financial aid from a grant/fellowship, or a work study 
appointment. When we asked students how they had learned about the various sources of financial 
aid, we learned that Upward Bound students and those in the control group often obtained their 
information from different sources. Upward Bound increased the likelihood of students receiving 
financial aid information from colleges and reduced their reliance on information provided by the 
military. 
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c. Upward Bound Students are More Actively Engaged in College Activities 

Many educators believe that retention in college is associated with students' successful 
integration into the college experience. When we asked students about the number of times they 
talked with faculty, met with an academic advisor, had informal contacts with faculty, or participated 
in study groups or school clubs we found that Upward Bound students were more likely to be 
engaged in such activities (see Table III.6). For example. Upward Bound led to a 30 percent increase 
in informal contacts with an advisor or faculty. Furthermore, Upward Bound students were 20 
percent more likely to have talked with faculty in their offices and participation in study groups or 
school clubs with other students. 

D. IMPACTS FOR SUBGROUPS OF STUDENTS 

In this section we focus on specific groups of students to assess who benefits most from Upward 
Bound. The groups we examined were students with lower/higher educational expectations; boys 
and girls; African American, Hispanic, and white youth; students who were from low-income and 
first generation college families, low-income only, and first generation only families; students whom 
we identified as having strong/weak academic preparation as high school freshmen; and students 
who were 9th and 10th grade Upward Bound applicants. Because we make a large number of 
comparisons in the subgroup analysis, we limited the outcomes to include students’ educational 
expectations, high school course taking, high school enrollment status, college attendance, credits 
earned in postsecondary schools, and college selectivity. 

Our exploration of subgroup impacts shows that Upward Bound had large impacts on some 
groups of students, particularly students with lower educational expectations, Hispanic and white 

, . •, '/V " 

students, boys, students from low-income/first generation and low-income only families, and 
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TABLE III.6 



IMPACT OF UPWARD BOUND ON COLLEGE ACTIVITIES 





Ail Students 




Participants 




. Treatment 


Control : 








■ Mean 


Mean 


Impact ' 


■ Impact' . 


College activities: (Number of times in the last 
term attended college) 










Talk with faculty in their offices 


0.6 


0.5 


O.l* 


0.1* 


Meet with advisor about academic plans 


0.6 


0.5 


0.1 


O.l* 


Have informal contacts with advisor or faculty 


0.6 


0.4 


0.2* 


0.2* 


Participate in study groups with other students 


0.7 


0.6 


O.l* 


0.2* 


Go places with friends from school 


0.9 


0.8 


0.1 


O.l* 


Participate in student assistance centers 


0.4 


0.3 


0.1 


0.1 


Participate in school clubs 


0.3 


0.2 


O.l* 


O.l* 


Attend career-related lectures with friends 


0.3 


0.2 


0.0 


0.0 


Participate in sports, music, drama 


0.2 


0.2 


0.0 


0.0 


Cut classes 


0.2 


0.2 


0.1 


0.1 



•Significant at the .10 level using a one-tailed t-test. 
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students who had earned fewer credits in core subjects in 9th grade and had a lower grade point 
average (more academically at-risk students).'^ 

1. Upward Bound Has Large Impacts on Students with Lower Initial Educational 

Expectations 

Before they applied for Upward Bound, almost 80 percent of the students in the study expected 
to complete at least a four-year college degree.'^ However, the students who appeared to benefit 
most from Upward Bound were the 20 percent with lower educational expectations (see Table 111.7). 
particularly when we consider their high school course taking behavior and high school completion. 
Upward Bound had a large impact on high school course taking for students who had lower 
expectations. When we looked across all high school subjects, we found that Upward Bound led to 
an increase of almost three credits for these students; this was a 1 7 percent increase over what similar 
students would achieve without Upward Bound. We found no impact on total credits earned for 
students with higher expectations. The impact estimates also show that Upward Bound increased 
the chances of low expectation students graduating from high school— 65 percent of the students in 
Upward Bound had graduated from high school and 52 percent of the control group had graduated. 
Upward Bound had no impact on high school graduation or the chances of dropping out of school 
for students with higher expectations. 

The program’s impact on credits earned in English, math, science, social studies, and foreign 
language for lower expectation students ranged from a 19 percent increase in science to a 26 percent 
increase in social studies. Only for math credits did Upward Bound have a significant impact on 

'^The core subjects refer to English, math, science, social studies, and foreign language. 

'^We defined students as having higher expectations if they expected to complete at least a 
bachelor’s degree; students who expected to complete less than a four-year college degree were 
defined as having lower educational expectations. 
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TABLE III.7 (continued) 
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course taking for students with higher expectations. Looking at course taking from the perspective 
of whether students were completing the New Basics Curriculum, we found that Upward Bound led 
to an increase in the percentage of lower expectations students who completed this curriculum. 
Although the impacts for students with lower and higher expectations were not always significantly 
different, the findings suggest that there may be substantial differences between the two groups.'^ 
Consistent with these findings,, we found that Upward Bound had large impacts on college 
attendance for students with lower initial expectations (Table III. 8). Upward Bound led to a 6-point 
increase for students with lower expectations and a 12-point increase for attending four-year 
colleges. Furthermore, Upward Bound had a large impact on earning credits in four-year colleges: 
students with lower expectations who were selected for Upward Bound earned almost ten credits and 
those in the control group earned about three credits. Also, students selected for Upward Bound, on 
average, earned fewer remedial credits attending two-year colleges and they attended more selective 
colleges than students in the control group. We found no impact on the college-related outcomes 
for students with higher initial expectations. 



2. Upward Bound Has Substantial Impacts on Boys 

Fewer than one-third of the Upward Bound applicants were boys; however, the findings 
presented here show that boys often benefitted more from Upward Bound than did girls (see Table 
III.9). Boys in Upward Bound expected to complete almost .8 years more schooling than boys in 
the control group; for girls we observed a statistically insignificant impact of about .2 years. Upward 
Bound’s impact on high school course taking for boys was almost two credits-about 1 1 percent— and 
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”In part, the lack of statistical significance concerning differences in impacts may reflect the 
small sample of students with lower expectations— the number of treatments and controls with lower 
expectations was 296 and 203, respectively. The number of students with higher initial expectations 
in the treatment and control groups was 1,097 and 1,002, respectively. 
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IMPACT OF UPWARD BOUND ON COLLEGE ATTENDANCE AND CREDITS EARNED IN 
COLLEGE FOR STUDENTS WITH HIGHER AND LOWER EDUCATIONAL EXPECTATIONS 
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IMPACT OF UPWARD BOUND ON EDUCATIONAL EXPECTATIONS AND 
HIGH SCHOOL COURSE TAKING FOR BOYS AND GIRLS 
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TABLE 111.9 (continued) 
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for girls, the impact was essentially zero. When we examined each of the subject areas and the 
chances of completing the New Basics curriculum, we foimd a similar pattem--large impacts for 
boys and at most, small impacts for girls. Upward Bound also reduced the risk of dropping out of 
high school for boys, but not girls. 

Upward Bound also led to larger impacts on college going for boys than girls (Table III. 10). 
Upward Bound increased the percent of boys going to a four-year college by 4 points and increased 
the number of credits earned in four-year college by almost 4 credits. Furthermore, being selected 
for Upward Bound led to boys attending more selective colleges. For girls, we found that Upward 
Bound had no impact on ever attending college, on earning non-remedial postsecondary credits, or 
attending a more selective college. However, girls selected for Upward Bound took fewer remedial 
courses when attending two-year colleges. 



3. Hispanic and White Youth Benefit More from Upward Bound than African American 

Youth 

The three largest racial/ethnic groups participating in Upward Bound were African Americans 
(50 percent), Hispanics (22 percent), and whites (2 1 percent). When we compared the impacts for 
the three groups of students, we found some differences in their educational expectations, high 
school course taking, and college outcomes (see Table III.l 1). Upward Bound had a significant 
impact on the expectations of all three groups of students; about .4 years for Hispanic and white 
students and about .3 years for African Americans. Upward Bound’s impact on total credits earned 
in high school and by subject shows that often, white and Hispanic students experienced large 
impacts. For total credits earned while in high school. Upward Bound led to an increase of almost 
two credits for Hispanic and white students, which corresponds to about a 1 0 percent increase. With 
the important exception of earning credits in Advanced Placement/honors courses, we found no 
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IMPACT OF UPWARD BOUND ON COLLEGE ATTENDANCE AND CREDITS 
EARNED IN COLLEGE FOR BOYS AND GIRLS 



00 




# * 



VO O ^ P 

o o o 

I I 



I 



<N 

I 



— TT rn 

— o o o 

I t t 



* 



* 

00 O ^ P 

rn O O O 

t 



^ 

o o 



(N C> 



— Tt — 



* 

O 



m <N 

o o 

I ( 






^ oo 

(N — 



O — 



•— (N ^ 

iX O ^ O 



SJ. 0) O 
Sf 00 o 







o 

u 


o 

o 


C/) 






.2 




.2 






u. 


b_ 


A 














0> 

00 


A 

0> 

1 

Urn 

3 


eo 

o> 

6 


C 

.2 

CQ 

O 




0> 

00 

.2 

"o 


^ 

E .2 

!?-g 


o> 

00 

*0 


E .2 

!?-g 


C/) 


o 

U 


£ 




o 

> 


•o 

0> 


U 


1 ^ 


U 


1 " 


3 


•o 


•o 


•o 


•o 


c 










CQ 

C/3 


c 

0> 

ts 


c 

a> 

C 


c 

a> 

C 


c 

a> 

C 


k* 

CQ 

UJ 


o> 

> 

1 


"c5 "c5 
o o 


0> 

> 

1 


"c5 *5 
o o 


*o 

O 

•C 


< 


< 


< 


< 


(/) 

0> 


k. 

3 

O 

u. 




o 


H H 


o 

c/3 










k* 

U 











* 



* 



o 



<N 

O 



O 



u 

•Si 

0> 

CT) 

0> 

00 

"o 

U 



o 



o 

>> 



•O 

CQ 

•O 

s 

o. 

0> 



CO 

u 



* 

< 



cr. 



CL 

3 

2 

00 

•O 

3 

CO 

0> 



o. 



CO 

CO 




67 



IMPACT OF UPWARD BOUNI5 ON FDUCATIONAI. I:XPI;CTATI0NS AND HIGH SCHOOL 
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TABLE III. 1 1 (continued) 
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positive impacts for African American students. On average, African American students earned 1 6 
percent more credits in Advanced Placement/honors courses because of Upward Bound. 

For two of the five core subjects—math and science--we found large differences in impacts 
among the three racial/ethnic groups. The impacts for Hispanic youth ranged from a 9 percent 
increase in science credits to a 22 percent increase in math credits; for white students there was about 
a 1 3 percent increase in math and science credits. Given these large impacts, it is not surprising to 
find that Upward Bound had large impacts on the percent of Hispanic and white youth who 
completed the New Basics curriculum. For Hispanics, Upward Bound led to more than a 90 percent 
increase in the percentage completing the curriculum and for whites we found more than a 30 percent 
increase. Similar findings pertain to students’ high school status; that is. Upward Bound reduced 
the chances of white and Hispanic students dropping out of high school and had no impact on 
African American students. 

Upward Bound had a large impact on non-remedial credits earned in four-year colleges for 
Hispanic students, but no impacts for whites or African American students (Table III. 12). For white 
students, however Upward Bound led to earning fewer credits in remedial courses. Similarly, 
African-American youth who were selected for Upward Bound earned fewer credits in remedial 
courses while attending two-year colleges. We also found that Upward Bound led white students 
to attend more selective colleges, but Upward Bound had no impact on college selectivity for 
Hispanics or African Americans. 



4. Giving Students the Opportunity to Participate in Upward Bound Had Substantial 
Impacts on Low-income/First Generation Students 

According to the federal Upward Bound regulations, at least two-thirds of all program 
participants must be from low-income families andh^ potential first generation college students; the 



ERIC 



70 



99 



IMPACT OF UPWARD BOUND ON COLU-CiF A Tl liNDANCF AND CRI-DH S I- ARNI;D 
IN COLLFGI- FOR African American, WIIITF, AND HISPANIC STUDI-NTS 
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remaining participants must either be from a low-income family or be a potential first generation 
college student. More than 80 percent of the students participating in the study met both criterion, 
and the remaining students were mostly from first generation only households. Comparison of the 
impacts for the three groups of students (low-income/first generation, first generation only, and low- 
income only) showed that most of the impacts were concentrated in the low-income/first generation 
and low-income only groups (see Table III. 13). 

For students who were from low-income/first generation households. Upward Bound had 
substantial impacts on educational expectations and the courses they took in high school. The 
program’s impact on students’ expectations was more than one year for students from low-income 
only households and nearly one-third of a year for students from low-income/first generation 
households. Among students from first generation only households, we found no significant impact 
on students’ expectations, but substantial impacts on their parents’ expectations— as reported by 
students. 

Being selected for Upward Bound led to large gains in high school credits for students from 
both low-income/first generation households and low-income only households. The program’s 
impacts on total credits earned for the low-income/first generation and low-income only students 
were 1 and 2 credits, respectively. This means that being selected for Upward Bound increased low- 
income only students’ high school course taking by 12 percent; it increased the course taking of 
students from low-income/first generation households by 5 percent. Upward Bound also reduced 
the chances of dropping out of high school for low-income/first generation students and low-income 
only students, and had no impact on first generation only students. 

Upward Bound had a large impact on earning credits in Advanced Placement/honors courses 
for students from low-income only households. Giving such students the opportunity to participate 
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IMPACT OF UPWARD BOUND ON COLl.FGl-: F,NR01,1,MI:NT AND RF.I.A 1T:D OUTCOMI-S 
FOR Low-incomc AND PO'I FNTIAL FIRS I Cil-NFRA TION COl.UXil- S 1 UDI-N I S 
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TABLE III. 13 {continued) 
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in Upward Bound led to their earning three times more credits in these courses (.7 to 2.2). The 
impact of Upward Bound on the percentage of students completing the New Basics curriculum is 
confined to the low-income/first generation group. For this group, being selected for Upward Bound 
led to almost a 5 percent increase in the chances of completing the curriculum. 

With only one exception, we found that for college attendance, credits earned, and college 
selectivity. Upward Bound had no impacts for any of these groups of students (Table III. 14). The 
exception to this was that. Upward Bound reduced the number of remedial courses low-income/first 
generation students took and completed while attending a two-year college. 

5. Upward Bound Has Large Impacts on Students Who Have Lower Academic Performance 

as High School Freshmen 

Although one of the eligibility criteria for selecting students for Upward Bound is that they have 
a need for academic support, the program serves students with a wide range of academic skills. For 
example, students who applied for Upward Bound and were near the top in academic performance 
had earned five credits as a high school freshman in the core subjects and had a B+ grade average; 
students at the lower end of the academic skills distribution had earned about two credits in the core 
subjects and had D average.'* Students who most benefit from Upward Bound, however, are often 
those with greater need for academic support. 

To compare impacts for students who had lower/higher academic performance as high school 
freshmen, we constructed an index of academic risk based on (1) credits earned in the five core 
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'*Students with high and low academic performance refer to those who were at the 90th and the 
10th percentile, respectively, of the distribution of credits earned in the five core subjects and overall 
high school grade point average. These estimates are based on students’ experiences in 9th grade. 
Some students entered Upward Bound as rising 9th graders and the program may have had a small 
impact on their scores; however, we believe that while these values may slightly overestimate 
students’ course taking and grade point average before entering Upward Bound, they provide a good 
indicator of the kinds of students who apply for the program. 
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subjects and (2) students’ grades in high school.” Based on the index, we ranked students as being 
at low risk or high risk for school failure. The average student we considered most at-risk had 
earned three credits in the five core subjects and had a C average. Students who were less at risk had 

earned almost five credits and had a B- average. 

Although Upward Bound had similar impacts on students’ educational expectations for the 
high/low risk groups (about .3 years), the results in Table III. 15 show that substantial impacts were 
present for students’ course taking. For the most at-risk group, we found that being selected for 
Upward Bound led to increases in all the core subjects except science. On average, we found an 
impact of about 1 .3 credits across the five subjects-more than a 1 3 percent increase. We also found 
that Upward Bound had large impacts on earning credits in Advanced Placement/honors classes and 

■; on completing the New Basics curriculum for students who were not most at-risk; we observed no 
impacts for other students. Among the outcomes showing students’ high school completion status, 

V V 

“ • we found that Upward Bound reduced the chances of dropping out for both groups, but much more 

\ * i 

so for those who were most at-risk; Upward Bound participation reduced the dropout rate by 6 
points for this group. 

Upward Bound appears to have only a few impacts on college outcomes, despite students’ at- 
risk status (Table III. 16). Upward Bound students in the lower half of the academic skills 
distribution were more likely to attend four-year colleges and earned about two more non-remedial 



”The index was formed on the basis of a factor analysis and the predicted first principal 
component extracted from the factor analysis. After we computed the principal component score 
(index) for each student, we separated students into two groups—students who had a score above the 
typical student (median) and those who had a score below the typical student. Students with a higher 
score were labeled as less at risk and those with a lower score were labeled as most at risk of school 
9^- failure. 
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IMPACT OF UPWARD BOUND ON STUDENTS’ EDUCATIONAL EXPECTATIONS AND 
HIGH SCHOOL COURSE TAKING, BY STUDENTS’ AT-RISK STATUS 
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TABLE 111.15 (continued) 
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IMPACT OF UPWARD BOUND ON COLLEGE ATTENDANCE AND CREDI I S 
EARNED IN COLLEGE BY STUDENTS’ AT-RISK STATUS 
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‘Impact on school status outcomes expressed as percentage points. 

‘'This test of no interaction shows whether we found a substantial difference in the impacts for the subgroups. A * indicates that the probability 
of rejecting the hypothesis of no difference in impacts by chance was . 1 0 or less. 



college credits than similar students in the control group. For students with stronger academic 
performance as a freshman in high school, we found that being selected for Upward Bound led to 
them completing fewer non-remedial courses when attending two-year colleges, but the program had 
no impact on the other college outcomes. 

6. The Impact of Upward Bound Is Similar for 9th- and lOth-Grade Applicants 

Students from a wide range of grades may apply for Upward Bound; however, there is little or 
no information concerning when it is most beneficial for students to enter the program. To assess 
the impact of starting the program at different points during high school, we compared the 
experiences of the two largest groups of applicants--9th and 10th graders. We did not include 
.students who were still in 8th grade or who were in 1 1th grade when they applied in this phase of 
the analysis because there were relatively few of them in the study. The outcomes we used when 
-computing impacts for these groups were focused on high school course taking and students’ 
educational expectations. We did not use any of the college related outcomes because the 9th grade 
^applicants were too young to have attended college when we last collected data.’” 

The findings concerning Upward Bound’s impact on educational expectations and credits earned 
in high school were inconsistent. Among the younger applicants, we found that being selected for 
Upward Bound led to substantial impacts on students’ educational expectations and on the 
expectations of their mothers and fathers, as reported by students. For the older applicants we found 
that Upward Bound had no impact on educational expectations (see Table III.l 7). 



^°We excluded from the analysis all 9th graders who reported that they were in the 8th grade 
during the 1991/92 school year. When we compared the percentage of high school graduates in the 
9th grade and 10th grade cohorts, we found that even with this exclusion restriction, the percentage 
of 9th graders who had graduated was lower than for 1 0th graders--?? percent of the 9th graders had 
graduated by 1996 and almost 90 percent of the 10th graders had graduated. This suggests that the 
--hh grade cohort used in this phase of the analysis was somewhat younger than the 1 0th grade cohort. 
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IMPACT OF UPWARD BOUND ON I IIGI I SCI lOOL OUTCOMFS FOR 9TI I AND 10 Tl I CiRADI-: APPLICAN I S 
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Comparing the program’s impact on non remedial credits earned, we found that Upward Bound 
had some significant impacts for the lOth-grade applicants and almost always statistically 
insignificant impacts for 9th- grade applicants. Although the impacts for 9th- grade applicants were 
not statistically significant, they were similar to those computed for the older students, suggesting 
that Upward Bound may have about the same impact, regardless of whether students enter the 
program as 9th graders or 10th graders. For example, the impact on total non-remedial credits 
earned for both cohorts was about .8 credits-an increase of about one course more than they would 
have completed in the absence of the Upward Bound. 

The findings concerning credits earned by the two groups suggests that, at least for this set of 
outcomes, little may be gained by serving students before they are in the 1 0th grade. Before one 
concludes, however, that projects should focus their resources on older students, it would be prudent 
to wait for results from the next round of data collection that will show the impact of Upward Bound 
on the program’s primary outcome—increasing college access. 




120 



83 



IV. THE IMPACT OF DURATION IN UPWARD BOUND ON 
EDUCATIONAL OUTCOMES 



One strength of Upward Bound is that it is a multi-year program. Most applicants enter the 
program in the 9th or 1 0th grade, giving the students an opportunity to be exposed to the program 
for up to four years. Although this relatively long exposure to Upward Bound gives the program an 
opportunity to make a substantial impact on participants, the typical Upward Bound participant 
remains in the program for only about 1 9 months; the most common Upward Bound experience 
includes attending the program for one summer and parts of two academic years. Coupled with these 
low levels of participation, we found that, overall. Upward Bound had a modest impact or no impact 
on outcomes such as educational expectations, high school course taking, and college going and 
credits earned in college. 

To assess whether this limited participation in Upward Bound may account for the few impacts 
we uncovered in the evaluation when looking at all students, we undertook analyses that compared 
the outcomes for participants who were exposed to Upward Bound for different lengths of time and 
for those who completed the program and those who did not complete the program. These 
comparisons show that longer exposure to the program was often associated with substantial 
impacts, including the following: 

• Participants who remained in Upward bound for more than two years generally earned 
more high school credits, particularly in the five-core subjects, and had higher 
educational expectations. 

• For older students, greater exposure to Upward Bound was associated with higher 
college enrollment rates. 
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Furthermore, we found that remaining in Upward Bound until the end of students* senior year in 
high school had no independent impact on their outcomes. 

A. ESTIMATING THE EFFECTS OF DURATION ON PARTICIPANTS’ OUTCOMES 

To compute the effects of duration on student outcomes, we used an approach that was based 
on (1) the statistical matching of Upward Bound participants with similar students in the control 
group and (2) the estimation of an analytic model.' A simple comparison of outcomes for 
participants with different levels of exposure to Upward Bound is not sufficient because students 
with specific characteristics, such as greater motivation or more parental encouragement to persist 
in school, may remain in Upward Bound for different lengths of time and distort the duration- 
student outcomes relationship. To compute the effects of duration on the outcomes, we needed to 
isolate the effects of duration from these other factors. 

To implement our approach, we first sorted participants into three groups based on how long 
they remained in Upward Bound. The three groups we used were: (1) participated for 1 to 12 
months, (2) participated for 13 to 24 months, and (3) participated for more than 24 months. Next, 
we formed a one-to-one match between participants in the three duration groups with similar 
students in the control group. The one-to-one match between an Upward Bound participant and a 
similar student from the control group provided two outcomes that we compared: (1) an outcome 
showing what the Upward Bound participant accomplished and (2) what the participant would have 
accomplished without Upward Bound (that is, the outcome for the student selected from the control 



ERIC 



'The statistical approach we used required that we assume that all relevant factors that influence 
both participants length of exposure to Upward Bound and the student outcomes are included in the 
statistical model. Since it is likely that some factors were not included in the analysis, the findings 
about the relationship between duration and outcomes are not as robust as the findings derived from 
the random assignment experiment reported in Chapter III. 
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group). Comparing the two outcomes within the three duration groups showed the average effect 
of Upward Bound on participants who remained for one year (1-12 months), two years (13-24 
months), and more than two years (more than 24 months). The outcomes analyzed in this chapter 
were the same we used when comparing impacts for subgroups as discussed in Chapter 111. 

Since participants who remained in Upward Bound for a longer period had somewhat different 
characteristics than those who remained only a few months, such as being older when they applied 
for Upward Bound and having parents who were more involved in school related activities (see 
Table C.l), we statistically adjusted the effects of duration so that each duration group had similar 
background characteristics. This means that the effects we show in this chapter were adjusted so that 
differences on selected background characteristics for participants did not distort the relationship 
between duration and student outcomes. Some background characteristics we adjusted for were 
grade at application, sex, race/ethnicity, educational expectations, parent involvement in school 
activities, participation in high school activities, misbehavior in school, and employment. We give 
a more detailed description of the matching and estimation process in Appendix C. 

Separate results are presented for students who entered Upward Bound in 9th grade and 10th 
grade;- these are the two most common grades that students are in when they apply for the program.^ 
We kept these groups distinct because the maximum length of participation in the program varied 
with the grade in which students entered Upward Bound. For example, a student who entered in the 



-For students’ high school outcomes we present results for both cohorts; however, for their 
college outcomes we only present findings for the 10th grade cohort because 9th graders were too 
young to have attended college when we last collected data. 



^We do not look at results for 1 1th graders because there are so few 1 1th grade applicants in the 
study. 
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fall of grade 9 could have participated in 



FIGURE IV.1 

DURATION IN UPWARD BOUND 





Duration in Months □1*12 □13-24 ■24-»' 

A 10 th grade applicant, however, could 50 




have participated for a maximum of three 



30 

years. The distribution of participants by 



■ 





9th 



10th 



IV.l. 



Grade At Application 



B. EFFECTS OF DURATION IN UPWARD BOUND ON STUDENT OUTCOMES 

1. Participants Who Remained in Upward Bound for Longer Periods Had Higher 
Educational Expectations and Earned More Credits in High School 

Among 9 th and 10th graders, we found that Upward Bound often had substantial impacts on 

participants who remained in the program for more than two years (see Table IV.l). And among 

those who entered the program as 10th graders. Upward Bound led to substantial impacts for 

participants who were in the program for more than a year. 

a. Ninth Graders 

With only one exception, the findings concerning 9 th graders suggest that short-term 
participation (less than two years) had no impact on participants’ educational expectations. In other 
words, students who remained in Upward Bound for less than two years had similar educational 
expectations to students who did not have an opportunity to participate in the program. However, 
for those participants who stayed in Upward Bound for more than two years, the program led to an 
increase in educational expectations of about 0.6 years. Similarly, Upward Bound increased parents’ 



TABLE IV. 1 

EFFECTS OF DURATION ON PARTICIPANTS' HIGH SCHOOL OUTCOMES 



• Durattoa ;v 


1-12 


13-24 




Months 


Months 


Months 



9th Grade Cohort 

Educational Expectations (years of schooling) 



Students’ 


-0.3 


-0.8 


0.6* 


Fathers’ 


0.3 


-0.1 


0.9* 


Mothers’ 


0.6* 


-0.1 


0.8* 


Credits Earned in High School 


English 


-0.2 


-0.3 


0.7* 


Social Studies 


-0.2 


-0.1 


0.3 


Math 


0.5* 


0.0 


1.2* 


Science 


0.1 


-0.4 


0.2 


Foreign Language 


-0.1 


0.1 


0.2 


Total five core subjects 


0.1 


-0.7 


2.6* 


Total non-remedial 


0.4 


-2.0 


2.3* 


A P/Honors 


0.9* 


0.6 


0.9* 


Total non-remedial and remedial 


0.5 


-1.9 


2.4* 


Satisfying New Basics 


5“ 


15 


17 


Grade Point Average 


0.1 


-0.3 


-0.1 


10th Grade Cohort 

Educational Expectations (years of schooling) 


Students’ 


0.5 


0.6 


0.4 


Fathers’ 


0.5 


0.2 


-0.1 


Mothers’ 


0.5 


0.0 


-0.3 


Credits Earned in High School 


English 


0.2 


0.5* 


0.2 


Social Studies 


0.3 


0.4* 


0.6* 


Math 


-0.9 


0.0 


0.6* 


Science 


0.2 


0.0 


0.6* 


Foreign language 


-0.1 


0.2 


0.4* 


Total five core subjects 


0.6 


1.2* 


2.3* 


Total non-remedial 


0.5 


1.4* 


3.3* 


A P/Honors 


1.4 


1.1* 


1.7* 


Total non-remedial and remedial 


0.8 


1.2 


3.2* 


Satisfying New Basics Curriculum 


18* 


20* 


20* 


Grade Point Average 


-0.2 


-0.2 


0.1 



UBSF0742, 47, 50 

‘Adjusted effect statistically significant at the .10 level using a one-tailed test. 
•Effects on Satisfying the New Basics Curriculum are expressed as percentage points. 
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educational expectations for their children, as reported by their students, by about 0.8 years (about 
0.8 years for mothers and almost 0.9 years for fathers). 

Upward Bound also led to an increase in the number of credits participants earned in high school. 
For the five-core subjects, 9th grade participants with more than two years of exposure to the 
program earned almost 2.6 more credits than those who did not participate. This amounts to 
participants completing an average of two to three more courses in subjects such as English, math, 
science, social studies, or foreign language because of their participation in Upward Bound. Looking 
beyond the five-core subjects, we also found that long-term Upward Bound participants earned more 
credits in Advanced Placement and honors courses (0.9 credits) and, overall, completed more 
remedial and non-remedial high school credits (2.4 credits). 

b. Tenth Graders 

Among 10th graders who participated for more than two years, we found results concerning 
course taking that were often similar to those of the 9th graders. Moreover, for this cohort of 
applicants we found that Upward Bound participants with more than one year of exposure to the 
program also benefitted; that is, students who participated for 13 to 24 months and the group who 
participated for more than 24 months each realized substantial gains. The findings also suggest that 
each additional year that students participate may have led to larger effects. For example, 10th 
graders who participated for 1 to 12 months earned about 0.6 more credits in the five-core subjects, 
those who participated for 13 to 24 months earned about one more credit (1.2), and those who 
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participated for more than 24 months earned more than two additional credits (2.3) because of their 
participation.^ 

2. Longer Exposure to Upward Bound Led to Higher Postsecondarx' Attendance Rates, 

Attendance at More Selective Schools, and Earning More Postsecondary Credits 

Our results show that lOth-grade applicants who stayed in the program for 1 3 to 24 months were 
more likely to attend college, particularly a four-year college, than if they had not participated in 
Upward Bound (see Table IV. 2). Participating in Upward Bound for 13 to 24 months led to a 14 
percentage point increase in attendance at four-year colleges. Furthermore, these participants 
attended more selective colleges than their counterparts and earned five more non-remedial credits 
from four-year colleges. Among those who stayed for 1 to 12 months we found that they earned 
fewer remedial credits from four-year colleges. 

Why did we find that participating for 13-24 months and not more than 24 months resulted in 
larger impacts on college outcomes? In part, this finding comes about because for many of the 10th 
graders who entered Upward Bound late in their sophomore year, the maximum exposure to the 
program was two years. Moreover, many of the 1 0th graders who were in the program more than 
two years were too young to have attended college when we last collected data. Until we collect data 
again in 1989/99, it will be unclear whether participants with greater exposure to Upward Bound 
have better college outcomes than students with less exposure to the program. 



‘*The effect estimated for participating in Upward Bound for 13-24 months for total credits 
earned is not statistically significant at conventional levels of significance; however, the pattern is 
consistent with other results and suggests that 1 0th graders with a moderate amount of exposure to 
Upward Bound may also benefit from the program. 
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TABLE IV.2 



EFFECTS OF DURATION ON PARTICIPANTS* COLLEGE 
OUTCOMES: lOth-GRADE COHORT 







Duration 






1-12 Months 


13-24 Months 


24+ Months 


School Status: 
Attend college 


-23 


15* 


1 


Attend four-year school 


-15 


14* 


7 


Attend two-year school 


-3 




-1 


Attend vocational school 


-7 


-1 


-6 


Credits Earned in College 








Four-year college 


Non-remedial credits 


-2.9 


5.5* 


-0.7 


Remedial credits 


1 

P 

bo 


0.1 


0.0 


Two-year college 

Non-remedial credits 


-0.1 


1.2 


0.0 


Remedial credits 


0.1 


-0.8 


0.0 


College Selectivity 


-0.2 


0.4* 


0.2 



* Adjusted effect statistically significant at the .10 level using a one-tailed test. 



C. COMPUTING THE EFFECTS OF COMPLETING UPWARD BOUND 



To compute the effects of completing the Upward Bound program on students’ outcomes, we 
used a procedure similar to that used to estimate the effects of duration. To implement our approach, 
we sorted Upward Bound participants into two groups: ( 1 ) program completers— students still in 
Upward Bound at the end of their senior year of high school and (2) noncompleters. Next, we 
formed a one-to-one match between completers and similar students in the control group and 
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noncompleters and similar students in the control group. The one-to-one match between participants 
(completers and noncompleters) and similar students in the control group provided two outcomes 
that we compared: (1) an outcome showing what the participant accomplished and (2) what the 
participant would have accomplished without Upward Bound (that is, the outcome for the student 
selected from the control group). Comparing the outcomes within the two groups of participants 
show the average effect of Upward Bound on participants who stayed with Upward Bound and 
students who participated, but did not stay until the end of their senior year of high school. 

D. EFFECT OF PROGRAM COMPLETION ON STUDENT OUTCOMES 

Analyses of program completion were restricted to students who were in 10th grade when they 
applied for Upward Bound. We focused on this group for two reasons. First, there were too few 9th 
graders who had completed Upward Bound when we last collected data. Second, the number of 
older students was too small to conduct meaningful analyses. 

The results shown in Table IV. 3 suggest that except for a few cases, “completing” Upward 
Bound has no effect on student outcomes; that is, completers had outcomes that were similar to those 
of noncompleters. The most noteworthy exception concerns students’ educational expectations: 
Upward Bound completers expect to complete about 0.7 years more schooling that noncompleters. 
Are these results inconsistent with the results concerning duration? Probably not. Instead, it shows 
that Upward Bound needs time to develop the requisite skills in participants and there does not 
appear to be a qualitative impact that results from completing the program. 
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TABLE IV.3 



EFFECTS OF COMPLETING UPWARD BOUND FOR lOTH GRADE APPLICANTS 



Outcome 


Effect of 

Upward Bound on . 
Noncompleters 


Effect of 
Upward Bound 
on Completers 


' Impact 


Students' e.xpectations 


-.26 


.43 


.69* 


Fathers’ expectations 


-1.10 


-.26 


.84 


Mothers’ e.xpectations 


-.68 


.07 


.75 


Attend college 


.27 

1 


.04 


-.24 


Attend two-year college 


-.04 


.01 


.05 


Attend four-year college 


.21 


■11 


-.10 


Attend vocational school 


.09 


■ ' ■ -.08 


-.18 


High school credits 








English 


.64 


.59 


-.05 


Social science 


.32 


.45 


.12 


Math 


.25 


.38 


.13 


Science 


.06 


.14 


.08 


Vocational education 


-.22 


.46 


.68* 


Foreign language 


.17 


.15 


-.02 


Computer science 


.01 


-.05 


-.06 


Other 


.27 


1.10 


♦ 

00 


Five core subjects 


1.49 


1.70 


.27 


AP/honors 


2.33 


.55 


-1.78 


All 


1.57 


2.39 


.81 


New basics 


.16 


.12 


-.04 


Credits earned in college 








Four-year 








Remedial 


.13 


.31 


.19 


Nonremedial 


13.09 


3.82 


-9.17 


Two-year 








Remedial 


-.19 


-.11 


.08 


Nonremedial 


.86 


.97 


.11 


Selectivity 


.57 


.08 


-.49 



■^Indicates that the probability of rejecting the hypothesis of no effect by chance was less than or equal to 

. 10 . 



13D 

o 

ERIC 



94 



V. SUMMARY AND CONCLUSIONS 



Upward Bound serves each year about 44,000 high school students who are economically 
disadvantaged and/or are potential first-generation college students; the intent of the program is to 
increase participants’ skills and motivation so that they are more likely to succeed in college. The 
first phase of the national evaluation of Upward Bound has drawn on data provided by Upward 
Bound project directors and staff, target school liaisons, parents, and students. Analysis of these data 
show that Upward Bound projects provide an intensive academic program during the summer and 
a more modest program during the school year. Services provided by Upward Bound projects 
include formal instruction in high school subjects such as English, math, science, social studies, and 
foreign language; tutoring; and counseling. Most applicants are rising 9th graders, high school 
freshmen and sophomores; and most of them have high educational expectations. Moore (1997) 
describes these findings in detail. 

A. SUMMARY OF FINDINGS 

Four major findings stand out in this report (see Table V.l). First, while Upward Bound gives 
students the opportunity to participate in the program for up to three or four years, nearly 40 percent 
of the program participants who were in our study left the program during the first year and we 
estimate only about 40 percent of those who began the program will be in Upward Bound at the end 
of 12th grade. Because many participants leave the program during the first year and others depart 
from Upward Bound as high school juniors and seniors, the typical student gets only about 19 
months of exposure to Upward Bound services. 
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SUMMARY OF FINDINGS AFTER SECOND FOLLOW-UP OF STUDENTS 
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applicants) 



Second, when we look across all students served by Upward Bound, we find that the program 
has small impacts on students’ high school course taking and educational expectations. Since many 
students we selected for Upward Bound were too young to have attended college when we last 
collected data, it is too early to make definitive statements concerning the program’s impacts on 
college attendance and related outcomes. Furthermore, our findings concerning college access and 
retention reflect the experiences of students who entered Upward Bound later in high school and do 
not represent the experiences of those who entered, for example, as 9th graders. So far, our findings 
suggest that Upward Bound did not have an impact on the percentage of students who attended a 
two- or four-year college, nor on the selectivity of the college attended. It did, however, lead 
students to earn more credits from four-year colleges and to become more actively engaged in some 
college activities, such as talking with college faculty and working in study groups. 

Third, Upward Bound had substantial impacts for some groups of students, such as boys, white 



and Hispanic youth, and students who were most at risk of academic failure— those with lower 
educational expectations and those who had earned fewer credits in the five core subjects as high 
school freshmen. Upward Bound students completed more credits in the five-core subjects than their 
counterparts in the control group, and they were more likely to attend college and earn more credits 
from four-year colleges. The findings also show that the impacts on completing high school courses 
were similar for the 9th- and lOth-grade applicants; this suggests that at least for this outcome, there 
may be no benefit to starting the program in 9th grade instead of 10th grade. 

Fourth, Upward Bound participants who remained in the program for more than two years often 
benefitted more from participation than those with less exposure to the program. Students who 
remained in Upward Bound for shorter periods, particularly 9th-grade applicants, often experienced 
smaller impacts or no impacts. 
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B. IMPLICATIONS OF FINDINGS FOR PROGRAM IMPROVEMENT 



Overall, Upward Bound’s impacts on students’ high school outcomes were small. Furthermore, 
it is too early to tell if the program leads to increased college attendance and completion. We found, 
however, that the program can have large impacts for some groups of students and for those who 
remained for longer periods. These findings have broad implications for program improvement and 
suggest the Upward Bound program and its projects may consider two issues for increasing the 
program’s impact on student outcomes. 



1. Retaining Upward Bound Participants 

The national evaluation showed that longer participation was associated with larger impacts, but 
only about one-third of the participants persisted for more than two years in Upward Bound. The 
most common reason that participants listed for leaving Upward Bound was to take a job. We 
suggest here, as we did in Myers and Schirm (1997), that one way to hold Upward Bound 
participants in the program is through the provision of employment opportunities that complement 
the design and curriculum of the Upward Bound programs. 

Fitting employment opportunities into the Upward Bound schedule, however, may pose many 
challenges for project staff since Upward Bound projects generally meet once per week during the 
school year and for five to seven weeks each summer in a residential setting. The summer is 
particularly challenging because students participate in formal courses or other scheduled activities 
for much of the day. As a consequence, there is little time available for students to work without 
reducing the time spent on scheduled activities. Incorporating employment opportunities into the 
program may be unappealing because it appears to dilute the program’s academic emphasis. 
However, if employment opportunities keep students in the program for an additional year or more. 
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then this short-term reduction in academic services should be more than made up in subsequent 



years. 

2. Serving More Students Who are At Greater Risk of Educational Failure 

The second issue concerns the concentration of impacts among students who were at greater risk 
of academic failure; these students were defined as having lower educational expectations, or as 
students who, as high school freshmen, earned fewer credits in the core subjects and had lower grade 
point averages than the typical Upward Bound applicant. We found that these students often 
benefitted more from Upward Bound than did other students. These findings suggest the program 
may achieve more substantial impacts by changing how students are recruited and selected for 
Upward Bound. For example, doubling the percentage of students with lower educational 
expectations in the program (20 percent to 40 percent) could increase the average impact of Upward 
Bound on credits earned in the five-core subjects by almost 50 percent.' 

Possible steps for identifying these students include the following: 

• Recruiting and selecting more students who have earned few credits in core subjects 
such as English, math, science, social studies, and foreign language 

• Recruiting and selecting more students with D and C grade point averages 

• Working with high school guidance counselors to identify students who have the 
potential to complete college, but have lower educational expectations such as expecting 
to attend only a two-year college or expecting to attend a four-year college, but not 
expecting to complete a bachelor’s degree 



'We computed the expected increase in impacts by assuming that Upward Bound’s impact on 
individual students would remain constant even when there is a shift in the composition of students 
participating in Upward Bound. Carrying out such a plan would require working with youth who 
are even more at-risk of school failure than current applicants and maintaining the same basic 
program of instruction and services already being offered. 



Project staff already recruit some students with these characteristics; however, they do not recruit 
them in large numbers. For example, among the projects who participated in the impact study, fewer 
than 10 percent of the projects had 40 percent or more of their applicants report they had lower 
educational expectations. 

Some project directors have expressed concern about adjusting the mix, claiming that part of 
the treatment effect we observed for students with lower expectations was related to the presence of 
students in the project with higher educational expectations. Although our statistical analyses have 
shown no relationship between project impacts and the percentage of participants in a project with 
lower expectations, the effects of changing the mix should be monitored carefully. One approach 
for testing this shift in recruitment and selection policy is to set up demonstration projects that enroll 
different concentrations of students with lower expectations or poorer academic records, but provide 
the same services, and to compare the impacts under the alternative conditions. 

C. ISSUES TO BE CONSIDERED IN PHASE II OF THE NATIONAL EVALUATION OF 

UPWARD BOUND 

Phase I of the national evaluation followed a group of students who entered Upward Bound in 
the 1992/93 and 1993/94 school years. Most of these students had completed high school or just 
entered college when we last collected data. The U.S. Department of Education (ED) has awarded 
a contract to MPR to follow this group of students for five more years-through 2002-to assess the 
program’s impact on a series of college related outcomes, such as access, retention, and completion; 
selectivity of the colleges attended; college major; and employment related outcomes. Reports 
describing program impacts will be available in 1999 and 2001 . Besides continuing the evaluation 
along the same lines as Phase I, ED has ftmded two studies linked to the ongoing evaluation. The 
first is an assessment of the practices used by nine Upward Bound projects that have potentially 
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large impacts on key outcomes such as educational expectations and credits in the five core high 
school subjects. MPR staff and its subcontractors will visit these projects twice: once during the 
school year and another time during the summer program in 1998. The final report describing the 
findings from the case study visits will be available in early 1999. The second new study is an 
evaluation of the effects of Math/Science Centers (MSCs), funded under the Upward Bound 
Math/Science Initiative, on students’ outcomes, such as college major. This evaluation vvdll compare 
the experiences of a sample of MSC participants with those of similar students who are already part 
of the national evaluation (that is, participants in regular Upward Bound and students who were 
selected for the control group). We will include the sample of MSC participants in all future data 
collection efforts undertaken for the national evaluation. Reports describing the effects of MSC 
participation on student outcomes will be available in late 1999 and 2001. 
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Appendix A 



Sample Design, Weighting 
AND Error Estimation 




14 V 



In this appendix, we describe how the impact study sample was selected, how the baseline and 
second follow-up data were weighted, and how standard errors of sample estimates were calculated 
to reflect the sample design. 

A. SAMPLE DESIGN 

For the impact study, we selected a nationally representative sample of eligible Upward Bound 
applicants in two stages. First, we selected a nationally representative sample of Upward Bound 
projects to serve as “primary sampling units” (PSUs). Second, we selected eligible applicants to those 
projects and randomly assigned students to treatment and control groups. 

1. First Stage Sampling: Selection of Projects 

The “universe” of projects for the impact study-the collection of projects whose students are 
targeted for study and eligible to be selected for the study sample-consists of active regular projects 
that are (1) located in the 50 states and the District of Columbia; (2) hosted by postsecondary 
educational institutions; (3) mature, having operated for at least three years by October 1992; and (4) 
not serving only students with physical disabilities. Veterans projects and math/science projects are 
not considered regular projects. During the period when the impact study sample of students was 
being selected (roughly May 1992 through March 1994), there were 395 Upward Bound projects that 
met the definition of the universe.' 

From the universe of 395 projects, we selected a sample of 70 projects using stratified simple 
random sampling; each project in the universe was assigned to a group of projects (a stratum) and a 
sample was drawn from each stratum. Sampling rates varied across strata, so some projects had a 



'Some projects funded in the 1989-92 grant cycle were defunded in the 1992-95 grant cycle and 
therefore eliminated from the universe. Projects newly funded in the 1992-95 and later grant cycles 
were also excluded from the universe. 
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greater chance of being selected than other projects. Stratification with disproportionate sampling 
(unequal sampling rates) was used to ensure that enough projects and, therefore, enough students were 
selected to support precise estimates for relatively small, but important analytic subgroups, such as 
students in large projects or students in projects hosted by two-year postsecondary institutions. 

Table A. 1 displays the 46 strata used to select projects in the first-stage sampling for the impact 
study. The table also shows, for each stratum, the number of projects in the universe, the number of 
projects selected for the sample, and the number of projects in which random assignment of students 
was carried out. Within each stratum, projects were selected using simple random sampling without 
replacement. Thus, although selection probabilities varied across strata, each project in a given stratum 
had the same chance of being selected. That chance equals the number of projects selected divided 
by the number of projects in the universe in that stratum.^ 

Strata are defined, in part, by cross-tabulating three stratifying variables: 

1 . Location of the host institution 

2. Type and control of the host institution 

3. Project size 

The location variable has two categories: (1) urban and (2) rural. A project is classified as urban if the 
host institution is located in a Metropolitan Statistical Area (MSA), as defined by the U.S. Bureau of 
the Census. The type and control variable has three categories: (1) public, four-year; (2) private, four- 
year; and (3) two-year. Type and control was ascertained from the 1990-91 Integrated Postsecondary 



^Three of the projects in the sample are “backups,” selected randomly from the same strata as three 
originally selected projects for which it was determined that random assignment would be 
inappropriate. Two of the three originally selected projects were operating under special 
administrative provisions, and the third project had, for several years, been unable to fill all available 
openings. These three projects that were replaced by backups are included in the universe counts in 
Table A.l. 
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TABLE A. I 



DISTRIBUTION OF UPWARD BOUND PROJECTS: IMPACT STUDY 



Number of Projects 



Sample 



Stratum 



Universe 



Selected Respondents^ 



Urban: four-year, public 

Small: 

African American^ 
Latino 
Other 
Medium: 

Asian 

Native American 
Latino 
Other 
Large: 

African American 

Latino 

White 

Other 

Urban: four-year, private 
Small: 

African American 
Other 
Medium: 

Asian 

African American 
Latino 
Other 
Large: 

Asian 

African American 
Other 



14 

4 
7 

5 
2 
9 

56 

25 

6 
2 
6 



8 

5 

4 
38 

3 

5 

2 

22 

3 



2 

I 

1 

2 

1 

2 
I 

3 

3 

I 

I 



1 
1 

1 

3 

2 
I 

1 

5 

1 



2 

I 

1 

2 

1 

2 
I 

3 

3 

1 

1 



1 

1 

1 

3 

2 

1 

1 

3 

1 
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TABLE A. 1 {continued) 



Stratum 


■'UBiverse 


Number of Projects 
Sample 

Selected Respondeuts* | 


Urban: two-year 

Small: 


Native American 


1 


1 


1 


African American 


9 


3 


3 


Latino 


3 


1 


1 


Other 


5 


1 


1 


Medium: 


Asian 


2 


1 


1 


African American 


10 


3 


3 


Other 


4 


1 


1 


Large 


3 


1 


1 


Rural: four-year, public 

Small: 


White 


6 


1 


1 


Other 


6 


1 


1 


Medium: 


Native American 


7 


3 


2 


Latino 


4 


1 


1 


Other 


30 


1 


1 


Large: 


African American 


5 


1 


1 


Other 


10 


2 


2 


Rural: four-year, private 


Small 


7 


1 


1 


Medium 


14 


2 


2 


Large 


4 


1 


1 
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TABLE A.l {continued) 



Stratum 


Uuiverse 


Number of Projects . 

Sample. 

Selected Respondents* 


Rural: two-year 

Small: 


African American 


4 


2 


2 


White 


5 


1 


1 


Other 


6 


1 


1 


Medium: 


African American 


5 


1 


1 


White 


8 


2 


2 


Other 


5 


1 


1 


Large: 


White 


3 


1 


1 


Other 


3 


1 


1 


Total 


395 


70 


67 



'Respondents are projects in which random assignment was carried out. 

^At least 50 percent of the students served by “African American projects” are classified as African 
American according to the 1990-91 Upward Bound performance reports. Native American, Latino, 
and white projects are similarly defined. (Native American includes Alaskan Native.) For Asian 
projects, at least 25 percent of the students served are classified as Asian or Pacific Islander. 
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Education Data System (IPEDS) Institutional Characteristics file. The project size variable has three 
categories: ( 1 ) small (60 or fewer students), (2) medium (6 1 to 99 students), and (3) large ( 1 00 or more 
students). Enrollment figures were obtained from the 1990-91 Upward Bound performance reports. 

Although some strata are defined entirely in terms of the location, type and control, and project 
size variables, many strata are defined by also taking into account projects' racial/ethnic compositions. 
At least 25 percent of the students served by “Asian projects” are classified as Asian or Pacific 
Islander. For a Native American (including Alaskan Native), African American, Latino, or white 
project, at least 50 percent of the students served are classified as members of the specified 
racial/ethnic group. Data on race/ethnicity were obtained from Upward Bound performance reports. 

When possible, projects were sampled proportionately by racial/ethnic composition within 
classifications based on the other three stratifying variables. Thus, differences by racial/ethnic 
composition in the overall rates at which projects were sampled are largely due to disproportionate 
sampling by, mainly, size and type and control. Small projects, large projects, and projects hosted by 
two-year postsecondary institutions were oversampled to provide adequate sample sizes for subgroup 
analyses. 

2. Second Stage Sampling: Selection of Students 

For each project selected in the first stage, we identified its main recruiting period(s)— typically 
spring 1993, fall 1993, or both— that fell during the student sample intake period for the impact study 
(roughly October 1992 to March 1994). All eligible students applying to Upward Bound during a 
project’s main recruiting period(s) were selected with certainty for the baseline impact study sample 
and subject to random assignment to treatment (Upward Bound) and control groups. The exceptions 
were students designated as “exempt” from randorn assignment and students who could have been 




A-8 

1d3 



randomly chosen as “givebacks.” We discuss these exceptions after describing how random 
assignment was conducted. 



a. Random Assignment 

When a project had completed recruiting for a given recruiting period, we selected eligible 
applicants at random to fill all available program openings. Eligible applicants not selected for 
Upward Bound and assigned to the treatment group were assigned to the control group or, more 
accurately, to a waiting list that could be used to fill certain future program openings. The waiting list 
and students selected from it, so-called “post-initial treatments (PITs),” are discussed in the next 
section.^ 

At least one round of random assignment was conducted in each of 67 projects.'* In 17 projects 
with more than one recruiting period, there were two or more rounds of random assignment. We 
conducted a total of 87 rounds of random assignment. 

Many Upward Bound project directors were concerned that the element of chance introduced by 
random assignment could severely unbalance the student composition of their programs. For example, 
it would be possible for an entire cohort of students to be from just one target school or all female. The 
former outcome could have seriously damaged relationships with target schools whose students were 



^As we discuss later, students designated as post-initial treatments will not necessarily be members 
of the treatment group for baseline or follow-up analyses. 

''As indicated in Table A. 1 , random assignment was not carried out in 3 of the 70 projects selected 
for the sample. The stated policy of one of those three projects was to serve all eligible applicants. 
Although not policy, the practice of another project was also to serve all eligible applicants because 
there were few eligible students attending the project’s target schools— just enough students to fill 
program openings, leaving none to form a control group. The third project had its funding cut and had 
no openings for new students. These three projects could not be replaced by backups even though, as 
noted earlier, three other projects in which random assignment could not be carried out had been 
replaced. Backups could not be selected because random assignment was determined to be infeasible 
only after it had been announced that no additional projects would be selected for the impact study. 
Failure to carry out random assignment in originally selected projects may introduce bias of unknown 
direction and magnitude into sample estimates. 



not selected while the latter might have hampered program operations if there were not enough 
dormitory rooms available for females during the summer session. Therefore, project directors were 
allowed to specify random assignment strata and, subject to there being enough eligible applicants, to 
allocate available program openings across the strata to obtain the desired mix of students.’ In all, 
there were 339 random assignment strata. Within a given stratum, random assignment was conducted 
as described before. The eligible applicants in the stratum were chosen at random to fill the available 
openings in the stratum. Students not picked for Upward Bound were assigned to the control 
group/waiting list. 

b. Exemptions, Givebacks, and PITs 

Exemptions. At the request of Upward Bound project directors, a very small number of students 
applying to Upward Bound were exempt from random assignment because it was determined that 
allowing the assignment of such students to be subject to the vagaries of chance could be unusually 
or permanently disruptive to normal program operations. For example, if a project and a local child 
protective services agency had a prior agreement that all eligible students referred by the agency would 
be accepted into Upward Bound, that agreement was not violated for the evaluation, and students 
referred during the sample intake period were exempt from random assignment. Strict policies of 
treating applicants from the same family the same were also honored and accounted for a few 
exemptions. For instance, one of a pair of twins applying to a project with such a policy was exempt 
from random assignment while the other twin was subject to random assignment. The exempt twin 
would be allowed to participate in Upward Bound only if the nonexempt twin was randomly assigned 



’Stratification was also needed in some instances to ensure that a project did not violate the federal 
requirement that two-thirds of the project’s students be both low income and potential first generation 
college. For projects with multiple rounds of random assignment, each round had its own set of strata. 
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to the treatment group. All exempt students are nonresearch cases, meaning they were excluded from 
all future data collection activities and from all analyses. 

Givebacks. Some projects recruited many more eligible applicants than were needed to fill 
available program openings and form a control group of adequate size. In such instances, we randomly 
selected students from the control group and “gave them back” to the projects immediately after 
random assignment. These givebacks could be selected by the projects to fill program openings when 
the impact study student intake period had ended. Although subject to random assignment, givebacks 
are not part of the baseline or follow-up samples. All 97 givebacks (distributed across 1 1 projects) are 
nonresearch cases. 

PITs. After being selected for Upward Bound, some students never enter the program. Other 
students enter, but leave before completing the program. Therefore, Upward Bound projects typically 
maintain waiting lists of students so that program openings can be filled without having to either mount 
a full-scale recruiting effort or wait until the next recruiting period. 

During the sample intake period for the impact study, projects were not allowed to have their own 
waiting lists~all nonexempt applicants were subject to random assignment.® To enable projects to 
maintain full enrollment under such conditions, we assigned students not selected for Upward Bound 
to an evaluation waiting list, rather than a strict control group. Students could be randomly selected 
from the evaluation waiting list to fill program openings, although such use of the waiting list was 
subject to time and size restrictions. Specifically, students could not be selected off the waiting list 
after a certain date, typically the start of the next recruiting period. Also, for a given random 
assignment stratum, a student could not be selected from the waiting list if the selection of a student 

®Even students who applied to Upward Bound and were placed on a project’s waiting, list prior 
to the sample intake period for the study were generally subject to random assignment. The only 
exceptions were students who had been previously promised admission when openings became 
available. Such students were among the small number of exemptions. 
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reduced the number of students remaining on the waiting list to less than about two-thirds the number 
of students originally assigned to the treatment group. Students randomly selected from the evaluation 
waiting list are designated as PITs. The next section discusses how PITs are used in baseline and 
second follow-up analyses. 

B. WEIGHTING 

Students were assigned sample weights that we have used in analyzing impact study data. A 
student's weight indicates how many students in the universe she/he represents. A student with a 
weight of four represents herselfrhimself and three other students that .were not selected for the sample 
(or did not respond to a survey questionnaire or other data collection instrument). 

Weighting has three purposes. First, weighting ensures that the sample “weights up” to the 
universe, producing correct totals (subject to sampling variability).’ Second, for purposes of 
estimation, weighting “undoes” the effects of disproportionate sampling so that two strata with the 
same number of students in the universe are counted equally even if they have different numbers of 
students in the sample. Third, weighting adjusts for nonresponse. 

In the following sections, we describe how we assigned baseline and second follow-up weights. 
We constructed separate second follow-up weights for analyzing data from the survey, high school 
transcripts, and postsecondary transcripts. To exclude exemptions and givebacks from all analyses, 
we assigned them zero baseline and second follow-up weights. In contrast, all PITs were included in 
baseline analyses and received nonzero baseline weights. Whether a PIT received a nonzero second 
follow-up weight depended on when that student was selected off the evaluation waiting list, as 
discussed below. 



’For example, without weighting, a total estimated from a simple one-in-two random sample 
would, on average, fall short of the true (population) total by 50 percent. 
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1. Baseline Weights 

We assigned nonzero baseline weights to 3,028 students-all nonexempt students except 
givebacks-divided into 1,479 treatments and 1,549 controls.® A student’s baseline weight is: 

1 {number of applicants) 

w ^ » 

project selection probability {number of applicants - number of givebacks)^ 



where s indexes the student’s random assignment stratum. This baseline weight is the inverse of the 
student’s probability of being selected for the baseline sample. That selection probability is: 



p = project selection probability x 



{number of applicants - number of givebacks)^ 
{number of applicants)^ 



The first term on the right side of this last expression is the selection probability for the project to 
which the student applied (adjusted for nonresponse, that is, failure to carry out random assignment). 
The second term is the selection probability for the student conditional on the student’s project having 
been selected and random assignment carried out. In other words, the two terms are, respectively, the 
first and second stage selection probabilities. Their product gives the student’s overall (unconditional) 
selection probability.’ 

The first and second stage selection probabilities are easy to calculate. The first stage probability, 
the “project selection probability” in the last expression, is the number of projects in which random 
assignment was carried out— given in the last column of numbers in Table A. 1— divided by the number 



*For the baseline sample, students were designated as treatments or controls based on their initial 
random assignment status. Students initially selected for Upward Bound are treatments, while students 
initially placed on the evaluation waiting list, including students who later became PITs, are controls. 



’In the expressions in the text, “number of applicants” is, more precisely, the number of 
nonexempt eligible applicants. Also, the number of applicants minus the number of givebacks is the 
number of treatments plus the number of controls. 
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of projects in the universe-given in the first column of numbers in Table A. 1 . This ratio is calculated 
within a project’s first stage sampling stratum. The second stage selection probability equals one for 
all but 239 students who applied to the 1 1 projects that received givebacks. Although they applied to 
projects that received givebacks, these 239 students are not themselves givebacks and, therefore, 
received nonzero baseline weights. 

Two simple examples illustrate how we calculated baseline weights. For an applicant to a large, 
rural project hosted by a private, four-year university, the project selection probability is 1/4. 
(According to Table A.l, random assignment was carried out in 1 of the 4 large, rural projects hosted 
by private, four-year universities.) If there were 7 other applicants (8 applicants in all) and no 
givebacks in the student’s random assignment stratum, the second stage selection probability equals 
1, and the overall selection probability equals 1/4x1 = 1/4. Therefore, the student’s baseline weight 
is 4 (the inverse of 1/4), implying that the student represents herself/himself and three other students 
who applied to projects in which random assignment was not carried out (because those projects were 
never selected for the impact study sample). Alternatively, if there were 4 givebacks instead of none, 
the second stage selection probability equals 4/8, and the overall selection probability equals 1/4 x 4/8 
= 1/8. Then, the student’s baseline weight is 8. We can think of the selected student as representing 
herself/himself, one giveback, and six other students (two applicants to each of three projects not 
selected for the impact study sample). 

In the first example (with no givebacks), each of the 8 students in the baseline sample gets a 
weight of 4, and, together, the 8 students weight up to 8 x 4 = 32 students. In the second example 
(with 4 givebacks), each of the 4 students in the baseline sample gets a weight of 8, while each of the 
4 givebacks gets a weight of 0. Together, the 4 students in the baseline sample weight up to 4 x 8 = 
32 students, the correct weighted total. 
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Our simple examples involve applicants to a large, rural project hosted by a private, four-year 
university. Although random assignment was carried out in the one such project selected for the 
impact study sample, project selection probabilities were adjusted when necessary for nonresponse, 
that is, failure to carry out random assignment. For instance, the adjusted selection probability for the 
three large, urban, predominately African American projects hosted by four-year, private universities 
is 3/22. Using that selection probability, rather than 5/22, in weighting allows applicants to the 3 
responding projects to represent applicants to the 2 nonresponding projects (as well as applicants to 
the 17 projects that were never selected in the first place).'® 

2. Second FoIIow-up Survey Weights 

As discussed in the previous section, 3,028 students received nonzero baseline weights. We 
wanted second follow-up survey data for all of these students and succeeded in obtaining at least 
partially completed survey questionnaires for 2,608 students— an 86 percent response rate." We next 



'“It is easy to see the consequences of not adjusting for nonresponse. Suppose all 22 large, urban, 
predominantly African American projects hosted by private, four-year universities had 45 applicants 
during the study intake period for 22 x 45 = 990 applicants in all. With random assignment carried 
out in 3 of the 22 projects, our baseline sample consists of 3 x 45 = 135 applicants. If we do not adjust 
for nonresponse, taking the project selection probability as 5/22 instead of 3/22, each student in the 
baseline sample has an overall selection probability of 5/22 x 1 = 5/22 (assuming no givebacks) and 
a baseline weight of 22/5. The 135 students in the baseline sample weight up to 135 x 22/5 = 594, 
which is short of the correct total (990) by the nonresponse rate (40 percent). If we adjust for 
nonresponse, those students weight up to 135 x 22/3 = 990, the correct total. 

The nonresponse adjustment allows applicants to the responding projects to stand in for applicants 
to the nonresponding projects by distributing weight from the latter to the former. Had random 
assignment been carried out in all five projects selected for the sample, there would have been 5 x 45 
= 225 students in the baseline sample, and each would have had a weight of 22/5. When random 
assignment was not carried out in two of the five projects, 2 x 45 = 90 students did not make it into 
the sample. So, we divided the total weight that those 90 students would have received (90 x 22/5 = 
396) equally among the 3 x 45 = 135 students who made it into the sample when random assignment 
was carried out in the three projects to which they applied. The adjusted weight for each of these 1 35 
students is the unadjusted weight plus the distributive share, or 22/5 + 396/135 = 22/3 (after 
simplification). 

"Item nonresponse, that is, failure to answer individual questions, created little missing data above 
and beyond that created by unit nonresponse. (Unit nonresponse is the failure to answer any questions.) 



discuss how we weighted students for analyses of the second follow-up survey data and how, in 
weighting students, we adjusted for unit nonresponse. We begin by describing how we designated 
students as treatments, controls, or nonresearch cases. 

a. Designating Students as Treatments, Controls, or Nonresearch 

Of the 3,028 students who received nonzero baseline weights, 1,524 were designated as 
treatments, 1,320 as controls, and 184 as nonresearch cases for second follow-up analyses. 
Designations were made as follows. All 1,479 students assigned to the treatment group at initial 
random assignment are treatments for second follow-up analyses. Likewise, all 1,320 students 
assigned to the evaluation waiting list at initial random assignment and not randomly selected off it (as 
PITs) are controls. Of the 229 PITs, 45 are treatments for second follow-up analyses, and the rest are 
nonresearch cases. 

A PIT was designated as a treatment if two conditions were satisfied. First, the PIT had the 
opportunity to begin participating in Upward Bound at essentially the same time (often the same day) 
as the “original” treatments in the PIT’s random assignment stratum. Second, the PIT did not replace 
a treatment who dropped out of Upward Bound (or never showed up). PITs satisfying these two 
conditions were designated as treatments because it is assumed that they would have been original 
treatments had the Upward Bound project director not underestimated the number of open slots that 
were available at the initial random assignment.'^ As noted before, all other PITs were designated as 
nonresearch cases. 



'^Project directors often do not regard a slot as open until there is strong evidence that a previously 
enrolled student has dropped out. Therefore, rather than delaying student selection until the “last 
minute,” some slots that were later confirmed as open were not filled in the initial random assignment. 
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b. Preliminary Weights 



Each of the 3,028 students with a nonzero baseline weight, whether designated as treatment, 
control, or nonresearch for second follow-up analyses, was assigned her/his baseline weight as a 
preliminary second follow-up survey weight. 



c. Final Weights 

To develop final weights that both facilitate comparisons of treatments and controls by weighting 
them up to the same totals and incorporate an adjustment for nonresponse, we calculated a set of 
control totals. Specifically, we summed the preliminary weights of all students (treatment, control, and 
nonresearch) in each random assignment stratum to obtain 309 control totals.'^ After deriving the 
control totals, we multiplied each student’s preliminary weight by the inverse of her/his propensity 
score. This score is an estimated probability that the student was a respondent.'" This product was 



'^Because of nonresponse, 30 of the original 339 random assignment strata “lost” either all of their 
treatments or all of their controls (but never both as it turned out) and were combined with other strata. 
Strata were combined based on propensity scores, whose estimation is described later in the text. We 
combined an empty stratum with a nonempty stratum based on the similarity of students’ propensity 
scores, as measured by the average squared difference in propensity scores between students in a given 
empty stratum and students in a nonempty stratum with which the empty stratum might be combined. 
The nonempty stratum with the lowest average squared difference was judged the most similar to the 
empty stratum. We did not combine strata across projects or from random assignments that occurred 
at widely separated points in time. Propensity scores were used to combine strata because they reflect 
a broad range of characteristics related to nonresponse and to the outcomes that are examined in the 
impact analysis. There were 42 respondents and 54 nonrespondents or nonresearch cases in the 30 
empty strata. 

‘"The probability was obtained from a logistic regression model that related, for all 3,028 students, 
a binary response variable to a large set of predictor variables. This set included 30 variables in all- 
stratifying variables, variables measuring student baseline characteristics, and interaction variables 
obtained by multiplying pairs of characteristics variables. The characteristics variables and interaction 
variables in the “best” logit model, that is, the model used to estimate propensity scores were selected 
using a forward selection procedure with a liberal inclusion criterion. 

A variable included in the model should have two properties: (1) it should be a good predictor of 
the propensity to respond and (2) it should be a good predictor of the outcomes of interest in the impact 
analysis. We tried to ensure satisfaction of the first property by using the forward selection procedure. 
We tried to ensure satisfaction of the second property by applying our substantive knowledge about 
what factors have been shown to influence education and education-related outcomes. This led us to 
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then ratio-adjusted to obtain a final weight, such that the final weights for treatments and controls 
summed (separately) to the control totals.'^ Final weights for all nonrespondents and nonresearch 
cases equal zero. 

The final weights are constructed so that the weighted distribution of treatments across random 
assignment strata (calculated using the final weights) is the same as the weighted distribution of 
eligible applicants across random assignment strata at baseline (calculated using baseline weights). 
Likewise, the weighted distribution of controls is the same as the weighted distribution of eligible 
applicants. Therefore, the weighted distributions of treatments and controls are the same.'* 



consider for possible inclusion in the model only some of the many variables measured at baseline. 
In fact, only 17 characteristics variables were considered, and only two were included in the model by 
the selection procedure used. Counting stratifying and interaction variables as well as characteristics 
variables, we considered over 130 variables for inclusion in the model. 

'*The ratio adjustment procedure works as follows. Suppose that the product of the baseline 
weight and the inverted propensity score sums to 1 00 across the treatment group respondents. Then, 
if the control total for the random assignment stratum is 1 20, we multiply, for every treatment group 
respondent, the product of the baseline weight and the inverted propensity score by the ratio 120/100 
= 1 .2 to obtain a final weight. After this adjustment, the final weights sum to 120, the control total for 
the stratum. 

'*In addition to the weights for the analyses of second follow-up data, we created weights for the 
analyses in Chapter II of Upward Boimd participation, completion, and persistence. Each of the 1,524 
students designated as a treatment received a nonzero weight, while all other students received weights 
of zero. The “participation” weights for the 1 ,524 treatments were constructed so that the weighted 
distribution of treatments across random assignment strata (calculated using participation weights) is 
the same as the weighted distribution of eligible applicants across random assignment strata at baseline 
(calculated using baseline weights). Thus, for treatments, the participation and second follow-up 
weights are constructed in the same way, except that the participation weights do not need to be 
adjusted for unit nonresponse because there was none. 



3. Second Follow-up High School Transcript Weights 

To construct nonresponse-adjusted weights for the second follow-up analyses of high school 
transcript data, we followed the same procedures used to construct weights for the second follow-up 
survey data. As with the survey, we wanted high school transcript data for all of the 3,028 students 
and succeeded in obtaining transcripts for 2,494 students— an 82 percent response rate. To obtain a 
high school transcript for a sample member, we needed to know the names of the high schools the 
student attended. Therefore, all of the 420 students who are second follow-up survey nonrespondents 
are also nonrespondents for the high school transcripts. In addition, we were unable to obtain high 
school transcripts for 1 14 of the second follow-up survey respondents. A student’s designation as a 
treatment, control, or nonresearch case is the same for all components (survey, high school transcript, 
and postsecondary transcript) of the second follow-up study. 

a. Preliminary Weights 

Preliminary high school transcript weights are the same as the preliminary second follow-up 
survey weights. 

b. Final Weights 

Final high school transcript weights were constructed using the same methods described for the 
second follow-up survey weights.” 



”Since there was higher nonresponse to the high school transcripts, a total of 34 of the original 
339 random assignment strata “lost” either all of their treatments or all of their controls. These 34 
strata were combined with other strata, as described earlier. There were 53 respondents and 58 
nonrespondents or nonresearch cases in the 34 empty strata. The logistic regression model used to 
predict the probability of response to the high school transcript differed slightly from the model used 
to predict response to the second follow-up survey. 
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4. Second Follow-up Postsecondary Transcript Weights ’ 

In the second follow-up survey, 641 students said they had attended some type of postsecondary 
institution. Using the name of this institution provided by the survey respondent, we succeeded in 
obtaining college transcripts for 551 of the 641 students--an 86 percent response rate. In the 
development of the postsecondary transcript weights, the purpose of adjusting for nonresponse is to 
distribute the weight of the 90 nonrespondents across the 55 1 respondents. These respondents stand 
in for themselves and for their college-going peers for whom we do not have postsecondary transcripts. 

a. Preliminary Weights 

Each of the 3,028 students with nonzero baseline weights was assigned a preliminary 
postsecondary transcript weight equal to her/his final second follow-up survey weight. 

b. Final Weights 

The final postsecondary transcript weight is equal to the preliminary weight for each of the 
2,387 students not reporting postsecondary attendance in the second follow-up survey.'* In contrast, 
for the 641 postsecondary attendees, final weights do not equal preliminary weights. The final 
weights for the 90 postsecondary transcript nonrespondents were set equal to zero. The final weights 
for the 551 respondents were adjusted to compensate for the missing 90, as described next. 

To derive control totals for the nonresponse adjustment, we summed (separately for treatments 
and controls) the preliminary weights of all postsecondary attendees in each project.” Then, we 



'*Thus, the final postsecondary transcript weight equals the nonzero second follow-up survey 
weight for the 1 ,967 survey respondents who did not report any postsecondary attendance and the zero 
second follow-up survey weight for the 420 survey nonrespondents. 

”There were too few college attendees to control to random assignment stratum totals. So, we 
controlled to project totals instead. 



multiplied each student’s preliminary weight by the inverse of her/his propensity score.’® The 
resulting product was ratio-adjusted to obtain a final weight, such that the final weights for 
treatments and controls reporting postsecondary attendance summed (separately) to the project-level 
control totals.^' 

The final weights are constructed so that the weighted distribution of treatments across projects 
(calculated using the final weights) is the same as die weighted distribution of eligible applicants across 
projects at baseline (calculated using baseline weights).^^ Likewise, the weighted distribution of 
controls is the same as the weighted distribution of eligible applicants. Therefore, the weighted 
distributions of treatments and controls are the same. 

C. CALCULATING STANDARD ERRORS 

Throughout this report, we present many estimates, such as percentages and means. These 
estimates are called “point” estimates because they are single values, as opposed to ranges of values. 
We can also obtain standard errors for these point estimiates. As its name implies, a standard error is 



^®The logistic regression model used to predict the probability of response to the postsecondary 
transcripts differed significantly from the models used for the second follow-up survey and high school 
transcript weights. The postsecondary transcript model was estimated only across the sample of 641 
college attendees. Due to the smaller sample size, we eliminated all 105 interaction terms that were 
allowed in previous models. Consequently, the “best” model included only 10 predictors, nine of 
which were stratifying variables. 

A minor exception to this property arises because there were five nonrespondents in projects that 
did not include any college-attending respondents. Since we could not make the nonresponse 
adjustment for these students within their respective projects, we distributed their preliminary weights 
across all of the other respondents, separately for treatments and controls, so that the sum of the 
preliminary weights equals the sum of the final weights across the sample of 641 college attendees. 

“Despite the fact that the second follow-up survey weights and not the baseline weights were used 
as preliminary weights, this statement is true because the second follow-up survey weights were 
controlled to Ae baseline distribution of eligible applicants. The issue mentioned in the preceding note 
causes small differences from the baseline distribution. 
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an estimate of the error in a point estimate, that is, an expression of our uncertainty. Typically, 
standard errors are used to construct “interval” estimates or “confidence intervals” that give a range 
of possible values. A “95-percent” confidence interval extends from two standard errors below the 
point estimate to two standard errors above the point estimate. Thus, when we estimate from baseline 
data that 1 1 percent of eligible Upward Bound applicants do not expect to attend college and the 
standard error for this estimate is 2 percent, the 95-percent confidence interval runs from 11 -2x2 
to 1 1 + 2 X 2, or from 7 to 1 5 percent. One interpretation of this confidence interval is that if we 
repeated our sampling and estimation procedures 1 00 times (drawing a new random sample each time), 
about 95 percent or 95 of the 100 confidence intervals that we construct will contain the true 
percentage of eligible Upward Bound applicants who do not expect to attend college. That true 
percentage is the percentage that would have been obtained if we had surveyed all applicants in the 
universe, rather than a sample of 3,028 applicants.^^ In our example, we are 95-percent “confident” 
that the true percent of applicants who do not expect to attend college lies between 7 and 1 5 percent. 
Of course, the true percentage either does or does not lie in that range. 

Instead of confidence intervals, we can derive “t-statistics,” which are closely related to 
confidence intervals. Dividing a point estimate by its standard error gives a t-statistic. If that t-statistic 
is less than two in absolute value (that is, between -2 and 2), we conclude that, at the 95-percent 
confidence (5-percent significance) level, the point estimate is not “significantly different” from zero; 
in other words, the observed difference from zero may be due entirely to the element of chance 
introduced by sampling. Determining whether the t-statistic is less than two in absolute value is the 



^^We estimate that there were nearly 22,000 eligible students who applied to Upward Bound 
during the impact study sample intake period. 
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same as determining whether the confidence interval includes zero. When the confidence interval 
includes zero, we are not confident that the true value is different from zero.^'* 

To estimate error and express our uncertainty accurately, standard errors must be calculated using 
methods that reflect how the sample was drawn. For almost all analyses, we used SUDAAN or 
STATA, two computer software packages that use the Taylor series linearization method to calculate 
standard errors based on the user’s coded description of the sample design. 



^''For purposes of illustration, we have described here a “two-tailed” test, where any estimate far 
enough from zero-either above or below-casts doubt on the hypothesis that the true value is zero. 
Since the aim of most analyses presented in this report is to determine whether Upward Bound has a 
beneficial impact and not whether it just has some impact, good or. bad, we have usually conducted 
“one-tailed” tests. With a one-tailed test, only an impact estimate far enough above zero supports the 
hypothesis that Upward Bound has a beneficial impact. Any other impact estimate-including a large, 
but negative estimate— casts doubt on that hypothesis. 



Appendix B 
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We estimated the impact of Upward Bound on student outcomes using three approaches. To 
compute the impact for all students who were selected for Upward Bound, we used subclassification 
analysis. Estimation of subgroup impacts for students who were selected for Upward Bound was 
based on an analytic model. Finally, to compute the overall impact and subgroup impacts for 
participants we used statistical model the specified the relationship between selection into the 
treatment and control groups, participation, and student outcomes. We describe each of these 
approaches below. 

A. SUBCLASSIFICATION ANALYSIS 

Since we used random assignment to construct the two groups, we would ideally compute the 
program’s impact by comparing the average for an outcome for the treatment group with the average 
for the control group. However, small differences may exist between the treatment group and the 
control group (see Appendix D). To compute the overall impact of Upward Bound on student 
outcomes when taking into account differences in their background characteristics, we selected 
subclassification analysis; this approach requires few assumptions concerning the structure of the 
data (see, for example, Rosenbaum and Rubin, 1983 and 1984). For the subclassification analysis, 
we formed groups of similar students and within these groups we computed the difference between 
the average outcome for students who were selected for Upward Bound and the average outcome 
for students who were in the control group. To arrive at the overall impact, we averaged the within 
group estimates. 

The process for constructing the groups and for computing program impacts included the 
following steps: 

1 . For each student, we computed the probability of being in the treatment group. The 
probabilities were predicted using a logit model that included as predictors items 



such as sex, race/ethnicity, grade at application, students’ educational expectations, 
and participation in high school activities. 

2. With the predicted probabilities for the treatment group, we constructed six 
categories, each with an equal number of students. We sorted students in the 
treatment group and the control group into the six groups based on their predicted 
probabilities. The idea is that students, who have similar predicted probabilities of 
being selected for Upward Bound, will be similar in terms of the characteristics used 
to predict the probabilities (Rosenbaum and Rubin 1983). 

3. Within each of the six groups, we computed the program’s impact by taking the 
difference between the average of an outcome for the treatment group and the control 
group. The within group impact shows how much being selected for Upward Bound 
effects similar students’ outcomes. Computing the average of the six impact 
estimates shows the overall impact of Upward Bound. 

4. We computed th^ variances of the overall impact estimates with the following 

expression: where F. is the within group variance andp^. is the 

proportion of students from the treatment group who were in the jth category. 



B. IMPACTS FOR SUBGROUPS 

We used an analytic model to compute subgroup impacts. The analytic model included the 
student outcome as the dependent variable, and as independent variables in the model we included 
treatment status (Upward Bound or control group), an indicator for the subgroup, and the estimated 
probability of being in the treatment group. The type of analytic model we estimated was based on 
the measurement of the outcome variable. For outcomes measured on an interval scale such as, years 
of schooling or high school credits, we used a regression model. For outcomes measured on a 0/1 
scale, such as college attendance, we used a logit model. The general specification of the analytic 
model was: 

;;=Po+P,7’+P2G,+P3(7xG,)+P/+e 



where T, G, and/* correspond to the treatment indicator, an indicator variable that shows which 




subgroup a student was in, and the probability of being in the treatment group; Pq, P, , P 2 , P 3 , and P^ 
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are parameters to be estimated, respectively; and e shows the effect of unmeasured factors on the 
outcome.' The two parameters of direct interest areP, andPj because they can be used to compute 
the impact of Upward Bound on the subgroups and test whether a statistically significant difference 
exists between the impacts. 

This approach was used instead of the subclassification analysis because some groups were 
small when we computed impacts for specific groups such as students with lower initial education 
expectations. To use the analytic models approach when estimating subgroup impact, we needed 
to assume that the relationship between the probability of being selected for the treatment group and 
the outcome was linear and did not differ for the subgroups. For the subclassification analysis we 
did not need to make this assumption. 

C. IMPACTS FOR PROGRAM PARTICIPANTS 

Both approaches for computing program impacts described in sections A and B provide 
information about program impacts for all students who were selected for Upward Bound. Since 
some students we selected for Upward Bound decided not to participate in the program, it is useful 
to ask about the impacts for those who showed up for services (about 82 percent of all students 
selected for Upward Bound participated). To compute the impact for participants we use a 
framework suggested by Angrist, Imbens, and Rubin (1996). 

We can form an impact estimate for participants by using the following framework. First, we 
express the mean (expected value) for an outcome such as academic credits earned in high school 
for students in the treatment group and the control group as: 

' This model specification permits an analysis of two subgroups such as boys and girls, or 
students with lower or higher initial educational expectations. The model can be expanded to 
incorporate more subgroups by including additional subgroup indicators and interaction terms. 
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Eiyr) =^(Vr I •^= 1 W5= 1 )r+-£(Vr I E=0)[\ -Pr(S= 1 )^] 



and 

E(y^) =E(y^ 1 5= 1 )Pr{S= 1 )c*E(y^ \ 5=0)[ 1 -Pr(S= 1 ) J 

vdiere E(y.j) and£(V(,) are the means for the outcome in the treatment group and the control group, £(y^| S = 1 ) 
and£(V(p|S=l) are the means for the treatment group and the control group for students who showed 
up for services or who would have shown up for services if they had been given the opportunity to 
participate in Upward Bound, £(y^|S=0) and£(y^|S=0) are the means for the outcomes for students 
in the treatment group and the control group who did not show up for services and whom we would 
expect to not show up if given the opportunity to participate, and£r(S=l)^ and£r(S=l)^ are the 
probabilities that students in the treatment group and the control group showed up or would show 
up for services if given the opportunity to participate. 

For treatment group students, we are interested in the expected value of the student outcome, 
given that students participated in Upward Bound when offered the opportunity to participate. For 
students in the control group we are interested in the expected value of the student outcome, given 
that the students would have participated if given the opportunity. Using the expressions we have 
already defined, we can express the desired quantities as: 

£(y^|S=l)=[£(y^)-£0;^|S=0)[l -£r(S=l V]]/£r(S=l)^ 
and 

£(yc|£=l)=[£(yc)-^(Vcl‘^=0)[l->P''(5=l)c]/£K5=l)c 
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Then, the impact for participants is: 

l(participants) =£(y j-| S= 1 ) ~E(y^ 15=1) 

If we assume that Upward Bound had no impact on the students who did not show up (that is, 
£(y^| S=0) =E(y(, \ S=0 ) ) and the probability of being a no-show in the treatment group and the control 
group are the same (that is, Pr(S=l\=PriS=\)^, then we can estimate the impact for participants 
as: 

I{participants) - [E{y-j) ~E{y(^yPr{S= 1 



As Angrist. Imbens, and Rubin (19xx) show. Bloom’s estimator is equivalent to the instrumental 
variables estimator when using the following setup: 



i/.=ao+a,r.+e./ 



and 






where d.^ T.^ and y. correspond to an indicator of whether a student participated in Upward Bound 
(0=no and l=yes), a treatment status indicator, and the outcome for the ith student, respectively; the 
a ‘s and P ‘s are parameters to be estimated. Here, an estimate of the parameter P, shows the impact 
of participating in Upward Bound on students’ outcomes. To adjust for possible differences in the 
characteristics of the students in the treatment group and the control group at baseline, we included 
in the two equations as an independent variable the predicted probability of being in the treatment 
group at the time of randomization. To compute statistical tests, we used the standard errors 
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estimated for the parameters in the model and computed t-tests. We extended this model 
specification to include subgroup indicators when we computed subgroup impacts for participants. 

Although the specific formulation for estimating impacts for participants is best suited for 
outcomes measured on an interval scale, we have applied it to binary outcomes as well (for example, 
whether a student went to college). Estimation of logit models or similar models in this context is 
not readily carried out. With the approach we applied here, we obtain unbiased estimates; however, 
they are less precise than if we had used an approach explicitly designed for analyzing binary 
outcomes. 
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We used a combination of statistical matching and modeling procedures to compute the effects 
of duration in Upward Bound on participants’s outcomes. Below, we describe each of these 
procedures. 

A. FORMING A ONE-TO-MATCH USING PROPENSITY SCORES 

Students who participated in Upward Bound were sorted into one of three duration groups: (1 ) 
participated for 1 to 12 months, (2) participated for 13 to 24 months, and (3) participated for more 
than 24 months. After sorting participants into the groups, we matched each participant with a 
similar student from the control group using a propensity score.' The propensity score corresponds 
to the estimated probability of a student being in Upward Bound for a specified amount of time (for 
example, 1 to 12 months). 

To estimate the propensity score, we took all participants from one duration group and pooled 
them with the students in the control group. A logit model was then used to predict each student’s 
propensity score. We coded the dependent variable for the logit model as 1 for students who 
participated in Upward Bound for a specific amount of time and 0 for those in the control group. 
The variables used to predict the chances of staying in Upward Bound included sex, race/ethnicity, 
educational expectations, years in the United States, misbehavior in school, employment status, and 
parental involvement in school related activities. Separate logit models were estimated for students 
in each duration group. 

After we computed the propensity scores, we then took each Upward Bound participant and 
matched them with a student in the control group who had the closest propensity score. This one-to- 
one match formed the basis for computing the effects of duration on participants’ outcomes. The 



' Rosenbaum and Rubin (1983) provide a detailed discussion of this approach. 
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outcome from the Upward Bound participant shows what the student achieved when attending and 
the outcome for the matched-control group student shows what the participant would have achieved 
if they had not participated in Upward Bound. Comparing the two outcomes shows the impact of 
Upward Bound for each student. 

B. ADJUSTING FOR DIFFERENCES AMONG DURATION GROUPS AND COMPUTING 

EFFECTS 

With the three groups of participants-students who remained in Upward Bound for 1 to 12 
months, 13 to 24 months, and more than 24 months— we can compute the impact of Upward Bound 
for students who remained in the program for each of these periods. Since participants who 
remained in Upward Bound for longer periods differed on a variety of observed characteristics which 
may have affected their outcomes (see Table C.l), we statistically adjusted for these differences 
when we computed the impacts for each duration group. To make these adjustments we used a 
regression model. The regression model was specified as: 

where d. is the difference in outcomes for participant i, shows if a participant attended an 
Upward Bound program for 13-24 months, D.^ indicates if a participant attended for more than 24 
months, X. is a vector of student characteristics, e . is factor that shows the effects of unobserved 
characteristics on the differences in outcomes, and the P‘s and a are parameters (or vectors of 
parameters) to be estimated. The adjusted impact of attending an Upward Bound program for 1 to 
12 months corresponds to ^^+Xa, the impact for participating for 12 to 24 months is P^+Pj+^a, 
and the impact of participating for more than 24 months is P 0 +P 2 where ^ is a vector of means 



O 

ERIC 



C-4 



17 8 



TABLE C.l 



BASELINE CHARACTERISTICS OF TREATMENT GROUP, 
BY DURATION IN UPWARD BOUND 





Months ini Upward Bound 




• 1-^2 


•!h3-24 


25 Months 


Baseline Variable 


; Months : 


r Months 


• nr More 


Students’ educational expectations 


16.7 


16.6 


16.6 


Fathers’ educational expectations 


16.6 


16.6 


16.7 


Mothers’ educational expectations 


16.7 


16.5 


17.1 


Grade at application 


9.4 


9.7 


9.0 


Number of siblings 


2.5 


2.7 


2.2 


Female 


0.70 


0.69 


0.69 


Race/Ethnicity: 








Black 


0.45 


0.43 


0.49 


Hispanic 


0.28 


0.20 


0.26 


White 


0.19 


0.29 


0.19 


Asian 


0.03 


0.04 


0.04 


Native American 


0.06 


0.04 


0.02 


Talked with parents sometimes/often about: 








Courses 


0.62 


0.81 


0.76 


School activities 


0.72 


0.78 


0.79 


Studies 


0.57 


0.66 


0.69 


Grades 


0.79 


0.81 


0.88 


Transferring to another school 


0.20 


0.26 


0.17 


Taking ACT/SAT exam 


0.28 


0.41 


0.31 


College plans 


0.77 


0.83 


0.85 


Parent did the following sometimes/often: 








Checked on homework 


0.68 


0.70 


0.76 


Helped with homework 


0.54 


0.55 


0.72 


Gave special privileges 


0.62 


0.61 


0.71 


Limited privileges 


0.60 


0.50 


0.59 


Required chores 


0.82 


0.87 


0.93 


Limited TV/video time 


0.46 


0.47 


0.63 


Limited time with friends 


0.69 


0.76 


0.71 
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TABLE C.l {continued) 



Months in Upward Bound 



Baseline Variable 


M2 

Months 


13-24 

Months 


25 Months 
or More 


Times: 


Late for school 


3.1 


2.3 


1.9 


Skipped classes 


0.8 


0.6 


0.6 


Missed a day of school 


4.2 


4.1 


3.1 


In trouble for not following school rules 


1.3 


0.8 


1.1 


Put on in-school suspension 


0.5 


0.2 


0.4 


Suspended 


0.3 


0.2 


0.2 


Transferred for disciplinary reasons 


0.0 


0.0 


0.0 


Arrested 


0.1 


0.0 


0.0 


Spent time in juvenile home 


0.0 


0.0 


0.1 


Parent: 


Attended school meeting 


0.61 


0.67 


0.65 


Spoke with teachers 


0.66 


0.77 


0.64 


Visited classes 


0.46 


0.45 


0.54 


Attended school event 


0.61 


0.57 


0.73 


Hours spent: 


On homework 


7.0 


7.5 


7.0 


On school-sponsored activities 


4.2 


4.2 


4.5 


Working (school year) 


1.4 


1.6 


0.6 


Working (summer) 


7.2 


6.9 


3.0 


Number of high school activities 


3.4 


2.8 


3.5 
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for the independent variables in the regression model. To compute tests of statistical significance 
we use the standard errors of the estimates.* 



^We have used the standard errors computed from a bootstrap procedure. The bootstrap is based 
on the idea of computing the direct variability in the parameter estimates of the regression equation 
when there is repeated sampling from the same population (for these analyses we used 200 bootstrap 
samples to estimate the variability in the estimates). The estimates of the standard errors only 
capture variability in the parameter estimates of the regression equation after the matches were 
formed. That is, we do not incorporate the variability that may have occurred in the matching 
process of different bootstrap samples that were used at two stage of the analysis. A consequence 
of not capturing variability in the matching process may be an underestimate of the standard errors 
and thus concluding that there were statistically significant effects when in fact there were none. 
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Appendix D 



Use of Supplemental Services 




182 



TABLED.! 

PERCENT OF STUDENTS USING SUPPLEMENTAL 
SERVICES: BY SUBGROUP 







Bound 








Any 


■■ ' Acftdemic; ■ 

Year ' Sumuier 


Any . . 




' Summer 


Overall 


41% 


39% 


15% 


58% • 


54% 


21% 


Educational Expectations: 


Higher 


44 


42 


16 


57 


54 


24 


Lower 


37 


36 


14 


64 


59 


17 


Race/Ethnicity: 


African American 


44 


42 


17 


63 


58 


28 


White 


40 


39 


12 


45 


43 


9 


Hispanic 


36 


34 


10 


55 


53 


16 


Sex: . 


Boys 


40 


37 


17 


56 


54 


14 


Girls 


42 


40 


14 


58 


54 


24 


Low Income/First Generation: 


LIFG 


41 


39 


15 


57 


54 


22 


LI 


67 


66 


34 


68 


67 


19 


FG 


34 


31 


10 


57 


52 


23 


Academic Risk: 


Lower 


43 


41 


14 


62 


56 


26 


Higher 


40 


38 


16 


53 


51 


16 
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